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1 .  EXECUTIVE  SUMMARY 


Divergent  Collaboration811  (DC)  is  a  new  approach  to  collaboration  that  uniquely  harnesses 
the  power  of  diversity.  It  is  a  capability  used  to  connect  individuals  from  a  wide  spectrum  of 
domains,  backgrounds,  and  areas  of  expertise  into  the  collaborative  innovation  process  in 
order  to  explore  a  technology  problem  in  a  new  way.  A  series  of  DC  workshops  were 
designed  and  conducted  by  the  Innovate,  Demonstrate,  Explore,  Apply  (IDEA)  Lab  at  the 
Wright  Brothers  Institute  (WBI)  in  tandem  with  a  team  of  Human  Factors  researchers  from 
the  Human  Effectiveness  Directorate  (RH)  at  the  Air  Force  Research  Laboratory  (AFRL). 
Together,  these  two  groups  of  people  formed  the  Executive  Team  for  the  DC  project.  The 
three  DC  workshops  focused  on  the  research  challenges  of  Resiliency,  Human 
Performance  Augmentation  (HPA),  and  Autonomy. 

The  workshop  format  was  based  on  WBI  TecAEdge’s  diverge/converge  paradigm  in  which 
people  spend  much  of  their  time  together  diverging  on  ideas;  exploring  the  problem  space  in 
different  ways.  This  is  followed  by  a  converge  activity  in  which  all  of  the  ideas  generated 
during  the  diverge  process  are  vetted  and  culled  in  an  effort  to  hone  in  on  the  most  relevant 
ideas  for  the  specific  problem  the  team  is  addressing.  Each  DC  workshop  lasted  a  day  and 
a  half  and  the  total  number  of  participants  for  each  workshop  ranged  from  18-22  people. 
Based  on  the  topic  of  the  workshop,  participants  were  chosen  in  fields  adjacent  to  the  area 
of  interest.  Prior  to  arrival,  participants  were  asked  to  complete  two  assessments;  the 
Dominance,  Influence,  Steadiness,  and  Compliance  (DiSC)  Personality  Test  and  the  Team 
Dimensions  Questionnaire.  Also,  two  questionnaires  were  developed  that  were 
administered  after  the  workshops.  One  focused  on  the  DC  process  and  the  other  focused 
on  the  products  of  the  workshop. 

The  next  paragraphs  review  the  results  of  these  workshops  from  three  perspectives: 
Process,  products,  and  research.  After  reflecting  on  the  three  workshops,  the  Executive 
Team  determined  that  the  biggest  lessons  learned  from  the  “process”  perspective  were:  1 ) 
trying  to  recruit  people  that  could  stay  in  the  uncomfortable  area  of  examining  the  problem 
space  for  longer  than  normal,  and  2)  determining  how  divergent  the  group  should  actually 
be. 

In  terms  of  products,  the  Resiliency  Workshop  netted  24  ideas  of  which  4  were  determined 
to  be  innovative/impactful  ideas  to  research  first.  The  HPA  Workshop  resulted  in  48  ideas  of 
which  4  were  thought  to  be  innovative/impactful  ideas.  And  finally,  the  Autonomy  Workshop 
generated  35  ideas  for  research  of  which  4  were  deemed  innovative  and  impactful.  These 
ideas  are  now  in  the  hands  of  the  problem  owners  to  pursue  them  as  they  desire. 

From  the  research  perspective,  the  DiSC  and  Team  Dimension  assessments  were  reviewed 
by  the  Executive  Team  for  each  group  of  workshop  participants.  In  terms  of  the  Resiliency 
Workshop,  participants  were  fairly  evenly  divided  among  three  of  the  four  Disc  dimensions 
[there  were  zero  participants  in  the  fourth  dimension]  and  all  four  of  the  four  Team 
Dimensions.  For  the  HPA  Workshop,  participants  were  fairly  evenly  divided  among  the  four 
DiSC  dimensions  and  the  four  Team  Dimensions.  However,  for  the  Autonomy  Workshop, 
the  majority  of  participants  fell  into  one  category  of  DiSC  and  only  one  category  for  the 
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Team  Dimension  as  well.  Having  this  imbalance  may  have  been  the  cause  of  some  of  the 
challenges  that  the  Executive  Team  experienced  when  conducting  the  Autonomy 
Workshop.  In  the  future,  these  assessments  will  be  used  prior  to  the  workshop  to  better 
balance  the  workshop  participants  if  possible  or  at  least  provide  some  information  to 
balance  the  smaller  team  breakout  sessions. 

Overall,  it  was  determined  that  DC  is  a  viable  process  that  will  foster  innovation  and  create 
new  ideas  that  might  not  be  found  using  the  traditional  methods  of  solving  problems. 
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2.  AN  INTRODUCTION  TO  DIVERGENT  COLLABORATION8" 


The  idea  of  DC  grew  out  of  an  informal  discussion  between  the  AFRL’s  Executive  Director 
(Mr.  Joe  Sciabica)  and  a  local  artist  (Mr.  Peter  Benkendorf)  at  a  local  coffee  shop.  They 
postulated  about  what  might  happen  if  technical  people  were  infused  with  artistic  people  in 
problem  solving  activities.  Then  they  put  this  idea  into  action. 

The  first  activity  that  ensued  was  the  Tech-Arts  Pilot  Project,  a  collaborative  innovation 
experiment  that  RH  and  the  IDEA  Lab  (teamed  with  Mr.  Benkendorf)  were  engaged  in 
throughout  201 1 .  This  project  was  an  attempt  to  introduce  some  non-traditional  thinking  into 
RH’s  information  visualization  technology  development  activities  by  facilitating  collaboration 
between  a  group  of  local  artistic  professionals  and  a  team  of  scientists  and  engineers 
(S&Es)  from  RH  as  well  as  from  AFRL’s  Sensors  Directorate  (AFRL/RY).  From  November 
2010  to  March  201 1 ,  these  two  groups  met  several  times  and  facilitated  brainstorming 
sessions  were  conducted  around  the  topic  of  information  visualization.  These  meetings 
produced  a  variety  of  innovative  approaches  to  information  visualization  and  resulted  in 
several  interesting  follow-on  projects  between  the  artists  and  RH  personnel.  Overall,  the 
combined  artist/S&E  team  concluded  that  collaboration  between  professionals  from  these 
two  different  but  related  domains  produced  innovative  ideas  that  should  be  further  explored. 

The  next  activity  teamed  students  from  Science,  Technology,  Engineering,  and  Mathematics 
(STEM)  backgrounds  with  students  from  Stivers  School  for  the  Arts  over  the  summer  of 
201 1 .  This  divergent  group  of  students  was  tasked  with  portraying  the  mission  and  vision 
statements  of  one  division  in  RH  in  an  innovative  and  creative  way  in  an  effort  to  give  more 
(non-technical)  people  a  better  understanding  of  the  organization’s  work.  Both  technical 
and  artistic  knowledge  was  necessary  for  this  project.  This  activity  resulted  in  a  number  of 
innovative  products. 

Both  of  these  activities  were  briefed  to  Mr.  Sciabica,  who  encouraged  members  of  the  IDEA 
Lab  to  continue  to  pursue  formalizing  the  process  so  it  could  be  used  for  future  activities  in 
an  effort  to  spur  innovation  and  creativity  when  addressing  current  AFRL  challenges.  The 
director  of  RH,  Mr.  Jack  Blackhurst,  defined  three  key  areas  for  the  DC  process  to  be 
applied  -  Resiliency,  HPA,  and  Autonomy.  These  three  areas  were  chosen  because  of  their 
importance,  complexity,  and  needed  innovation.  He  also  identified  three  “problem  owners” 
to  engage  with  during  the  workshop  design  and  execution.  An  Executive  Team  was  formed 
consisting  of  people  from  the  IDEA  Lab  and  the  RH’s  Warfighter  Interface  Division.  Also,  a 
process  consultant  was  added  to  the  group  shortly  thereafter.  The  mission  of  the  team  was 
twofold:  1 )  to  establish  a  DC  process  that  would  harness  the  power  of  combining  people 
from  different  perspectives  to  address  relevant  challenges,  and  2)  apply  that  process  to 
three  current  challenging  areas  in  the  Directorate,  refining  the  process  as  needed.  The 
objectives  for  the  members  of  the  Executive  team  varied.  The  IDEA  Lab  participants 
wanted  to  define  and  refine  the  DC  process  so  they  could  add  it  to  the  services  they  offer 
AFRL  researchers  to  facilitate  in  strategic  planning  and  problem  solving  activities.  The  RH 
participants  were  interested  in  looking  at  the  process  and  the  participants  from  a  research 
perspective,  seeing  how  the  characteristics  of  the  participants  impacted  the  process.  All 
members  of  the  Executive  team  wanted  to  produce  products  useful  to  the  problem  owners. 
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3.  THE  DIVERGENT  COLLABORATION5"  PROCESS 


The  DC  Executive  Team  consisted  of  Dr.  Bart  Barthelemy  and  Ms.  Candace  Treasure  of  the 
IDEA  Lab;  Dr.  Scott  Galster,  Dr.  Kristen  Liggett  and  2Lt  Rachael  Bryson  of  the  Warfighter 
Interface  Division,  Human  Effectiveness  Directorate.  Also,  Kathy  Hollingsworth,  a  process 
consultant,  was  added  to  the  group  to  help  guide  the  Executive  Team  in  workshop  design 
and  structure.  Appendix  A  contains  short  biographies  of  the  Executive  Team  members. 

This  team  conducted  a  careful  examination  in  the  following  areas:  current  innovation 
processes  being  employed,  potential  surveys  and  assessments  for  participant 
characterization,  metrics  to  measure  the  quantitative  and  qualitative  aspects  of  the  process, 
as  well  as  other  applicable  research  tools  and  techniques.  The  Executive  Team  worked 
collaboratively  to  design  the  DC  process,  which  took  approximately  3  months.  Activities 
during  this  time  fell  into  two  categories.  The  first  involved  selecting  participants;  the  second 
involved  designing  workshop  activities. 

3.1.  Participants 

3.1.1.  Selecting  Participants 

One  of  the  most  important  steps  in  DC  is  selecting  participants.  The  process  that  the 
Executive  Team  utilized  started  by  brainstorming  professions  and/or  areas  of  expertise  that 
were  adjacent  but  related  in  some  way  to  the  challenge  areas  (Resiliency,  HPA,  and 
Autonomy).  For  example,  AFRL  is  interested  in  determining  what  makes  a  human/machine 
team  effective  because  there  are  current  challenges  facing  remotely  piloted  aircraft 
operators  and  the  semi-autonomous  vehicles  they  operate.  These  challenges  may  be 
analogous  to  the  challenges  a  surgeon  faces  when  teamed  with  a  robot  to  perform  surgery. 
Next,  the  Executive  Team  went  through  an  objective  ranking  process  to  decide  which  career 
areas  were  most  important  to  include  in  the  workshop.  The  Executive  Team  then  identified 
individuals  that  fit  into  each  area,  either  through  personal  connections,  professional 
networks  and  associations,  or  internet  searches.  These  professionals  were  then  sent  an 
invitation  to  attend  that  included  a  flyer  highlighting  the  DC  process,  the  IDEA  Lab,  and  the 
facilitators  leading  the  effort.  Appendix  B  contains  a  copy  of  a  DC  workshop  invitation. 

3.1.2.  Assessing  Participants 

To  help  the  Executive  Team  better  understand  the  participants  for  each  workshop,  two 
assessments  were  administered  prior  to  the  workshops.  These  included  Inscape 
Publishing’s  DiSC  Classic  Profile  and  Team  Dimensions  2.0  Profile.  The  DiSC  measures 
personality  and  behavioral  style;  Team  Dimensions  measures  a  person’s  most  natural  team 
role.  These  assessments  were  intended  to  be  used  post-workshop  to  characterize  the 
participants  in  terms  of  their  communication  and  team  dynamic  styles  in  an  effort  to 
compare  the  participants’  attributes  to  workshop  execution  and  products. 
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3.2.  Workshop 
3.2.1.  Workshop  Model 

Central  to  DC  is  the  IDEA  Lab’s  Collaborative  Innovation  Process  (Figure  1).  The  Executive 
Team  used  this  process  as  a  model  for  developing  the  DC  process.  They  also  leveraged 
the  Tech-Art  Pilot  Project,  however,  Tech-Arts  was  a  three  month  activity  and  the  goal  for 
the  DC  workshops  was  a  more  time-condensed  workshop.  The  biggest  (and  most  time- 
consuming)  challenge  for  the  Executive  Team  was  converting  this  model  into  a  series  of 
meaningful  and  relevant  workshop  activities  to  harness  the  power  of  the  model.  The 
executive  team  met  regularly  to  design  and  construct  the  overall  DC  process,  as  well  as  a 
tailored  workshop  for  each  challenge  area. 


COLLABORATIVE  INNOVATION  PROCESS 


Problem 

Space 


Solution 

Space 


New  Solutions,  Opportunities,  and  Partners 


Figure  1.  IDEA  Lab’s  Collaborative  Innovation  Process 


3.2.2.  Workshop  Activities 

Although  the  main  activities  in  the  workshop  were  focused  on  divergent  thinking,  there  was 
also  a  need  to  include  trust-building  activities  and  group  events.  Each  DC  Workshop  was  a 
day-and-a-half  event  and  activities  varied  slightly  based  on  the  challenge  to  be  addressed. 
The  activities  were  specifically  designed  to  explore  the  problem  space  of  the  identified 
challenge  in  new  and  different  ways.  The  workshop  activities  were  comprised  of  three 
primary  categories:  trust  building,  diverging,  and  converging. 

3.2.2. 1  .Trust  Building 

One  of  the  factors  that  the  Executive  Team  believed  contributed  to  the  overarching  success 
of  the  Tech-Arts  Pilot  Project  was  the  amount  of  time  spent  establishing  trust  and  building 
relationships  between  the  two  communities  of  artists  and  engineers.  Similarly,  taking  the 
time  to  establish  trust  among  an  even  more  diverse  group  of  participants  was  a  critical 
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component  in  the  design  of  the  DC  workshops.  Trust  building  included  activities  to  get  to 
know  one  another  through  various  introduction  methods  (Two  Truths  and  a  Lie,  Introducing 
each  other,  etc.),  an  exercise  to  show  the  interrelatedness  of  the  group,  asking  participants 
to  explain  why  they  thought  they  were  invited  to  the  workshop  and  what  they  could 
contribute,  establishing  a  high-level  definition  of  the  challenge  area  (Resiliency,  HPA,  and 
Autonomy),  participating  in  an  exercise  called  Zoom/Re-Zoom  to  facilitate  appreciation  for 
other  perspective  and  problem  solving  skills,  and  a  mandatory  social  hour  for  networking. 


3. 2. 2. 2. Diverging 

Following  the  Diverge/Converge  model  (Figure  1 ),  the  Diverge  section  was  comprised  of 
various  brainstorming  exercises  to  get  participants  to  expand  and  extend  their  thinking  in 
multiple,  lateral,  and  different  directions.  These  activities  were  conducted  through  facilitated 
brainstorming  in  small  group  break-out  sessions,  which  were  then  presented  back  to  the 
larger  group.  Participants  were  exposed  to  the  problem  at  a  high,  surface  level,  without  the 
application,  context,  and  background  of  the  particular  Air  Force  problem  or  need. 


3. 2. 2. 3.  Converging 

Once  participants  thoroughly  explored  the  problem  space  through  the  diverging  exercises, 
they  were  then  exposed  to  the  actual  Air  Force,  or  AFRL  problem  and  context.  This 
transition  was  coined  by  the  Executive  Team  as  the  lens-shift,  and  served  to  focus  the 
group  on  the  actual  problem  at  hand.  Again,  through  facilitated  break-out  groups  and  large 
group  exercises,  the  participants  generated  new  ideas  and  also  discussed,  defined,  and 
refined  ideas  into  insights  that  might  influence  and  invigorate  new  and  existing  research  in 
AFRL.  These  ideas  and  insights,  the  product  output  of  a  workshop,  also  included  new 
directions  and  opportunities  for  AFRL  to  explore  further. 

3.2.3.  Measuring  the  Effectiveness  of  the  Workshop 

To  help  the  Executive  Team  determine  which  metrics  to  utilize  in  assessing  the 
effectiveness  and  success  of  the  DC  workshops,  the  team  consulted  Matt  Valenti,  Deputy 
Corporate  Development  Officer  for  AFRL,  in  the  preliminary  stages  of  this  effort. 
Measurements  in  the  form  of  surveys  and  guestionnaires,  centered  on  two  primary  areas: 

(1 )  the  DC  process,  and  (2)  the  products  of  the  workshops.  The  Process  Questionnaire  was 
taken  by  all  participants,  problem  owners,  and  Executive  Team  members.  This  survey 
centered  on  understanding  the  effectiveness  of  the  structure,  design,  and  process  utilized 
throughout  the  workshop,  as  well  as  to  identify  any  areas  of  improvement.  The  Product 
Questionnaire  was  given  to  the  problem  owners  and  the  Executive  Team  to  assess  the 
output  of  the  workshop  including  the  level  of  innovation  and  creativity. 


3.2.4.  Vetting  the  DC  Process  to  the  Problem  Owner 

The  next  step  in  the  planning  phase  was  to  contact  the  designated  AFRL  problem  owners. 
These  people  were  managers  or  researchers  working  in  the  technical  areas  of  Resiliency, 
HPA,  or  Autonomy.  The  Executive  Team  met  with  the  problem  owners  to  introduce  them  to 
the  DC  process  and  to  clarify  expectations.  Engagement  with  them  in  the  development  and 
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pre-workshop  planning  phase  was  designed  to  be  minimal  so  that  they  could  attend  the 
workshop  as  participants  -  objective  and  unbiased  in  exploring  the  problem  in  a  new  way. 
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4.  DIVERGENT  COLLABORATION5"  IN  ACTION 

This  section  will  highlight  the  details  of  the  three  DC  workshops  that  were  conducted. 


4.1.  The  Resiliency  Workshop 

4.1 .1 .  Resiliency  Challenge  Statement 

Every  day  the  men  and  women  in  our  armed  forces  are  placed  in  highly  stressful  situations, 
and  we  are  looking  at  ways  of  mitigating  the  pressure  placed  upon  them  by  exploring 
Resiliency. 

4.1.2.  Resiliency  Workshop  Participants 

After  considering  approximately  65  individuals  with  backgrounds  in  over  30  different  areas 
of  interest,  the  Executive  Team  ultimately  chose  twelve  collaborators  to  attend  the 
Resiliency  Workshop.  Their  backgrounds  included  the  following: 

•  Life  Coach;  Motivational  Speaker 

•  Military  Counselor 

•  Division  I  College  Football  Player;  Local  Community  Center  Director 

•  Personal  Trainer;  Physiologist 

•  Spiritual  Advisor;  Marianist  Brother;  Business  Ethics  Professor 

•  Emergency  Room  Doctor;  Emergency  Management 

•  Alternative  Medicine  Therapist 

•  Life  Coach 

•  Marine 

•  Hospital  Administrator 

•  Physical,  Occupational,  and  Speech  Therapist 

•  Qi  Gong/Tai  Kwon  Do  Instructor 

Additional  participant  demographic  information  can  be  found  in  Appendix  C. 

Figure  2  shows  participant  information  from  the  DiSC  and  Team  Dimensions  Profiles 
administered  pre-workshop. 
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Resiliency 


Key 

■  D(ominance) 

■  i(nfluence) 

■  S(teadiness) 

■  C(onscientiousness) 

Key* 

■  Creator 

■  Advancer 

■  Executor 

■  Refiner 


Figure  2.  DiSC  and  Team  Dimension  Summary  for  Resiliency  Participants 

4.1.3.  Resiliency  Agenda 
Day  1 


Time 

Event 

12:00-12:15 

Arrival  and  Check-In 

12:15-1:15 

Working  Lunch  -  Introduction  of  Participants 

1:15-2:15 

Defining  Resiliency  (Participants) 

2:15-2:30 

Break 

2:30-3:30 

Connecting  the  Dots 

3:30-4:00 

Overview  of  DC;  Resiliency  (Executive  Team) 

4:00-4:15 

Break 

4:15-5:45 

Diverge  Activity:  Metaphoric  Thinking 

5:45-6:30 

Social  Hour 
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Resiliency 


Day  2 


Time 

Event 

8:00-8:30 

Check-in  &  Breakfast 

8:30-8:45 

Previous  Day  Recap 

8:45-9:30 

Diverge  Activity:  Build  A  Life 

9:30-9:45 

Break 

9:45-11:45 

Diverge  Activity:  Show  and  Tell 

11:45-1:00 

Lunch 

1:00-3:00 

Converge  Activity 

3:00-3:15 

Break 

3:15-4:00 

Admin  (Ouestionnaire  Completion) 

4:00-5:00 

Closing  Recap 

4.1.4.  Resiliency  Products 

Appendix  D  contains  the  comprehensive  list  of  ideas  generated  by  the  Resiliency  group. 
Listed  below  is  a  summary  of  the  most  innovative  or  most  impactful  ideas  on  the  Resiliency 
challenge  according  to  the  Executive  Team: 

1 .  Continuity  of  Care  -  providing  warfighters  with  care,  support,  and  assistance  prior  to 
deployment,  during  deployment,  and  after  deployment. 

2.  Institute  Peer  Mentors  -  someone  who  has  “been  there,  done  that”,  rather  than 
military  chaplains  or  psychologists  that  not  only  carry  a  stigma  but  are  also  perceived 
to  be  disconnected  to  warfighter  experiences. 

3.  Take  a  holistic  approach  to  care  -  take  into  consideration  all  aspects  of  an 
individual’s  life  as  opposed  to  simply  their  mental  state.  This  might  include  spiritual, 
emotional,  physical,  occupational,  social,  as  well  as  mental  considerations. 

4.  Validation  of  the  problem  owner’s  conclusions  on  solution  opportunities  to  further 
explore  in  Resiliency.  Overall  the  Resiliency  problem  owners  felt  that  the  divergent 
collaborators  came  to  similar  conclusions  as  their  team  had  prior  to  the  Divergent 
Collaboration811  workshop.  This  validation,  that  the  insights  and  ideas  generated  by 
an  independent  and  objective  group  aligned  with  their  own,  was  just  as  powerful  and 
valuable  to  the  problem  owners  as  were  brand  new  ideas  and  approaches  to  the 
problem. 
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4.1.5.  Resiliency  Workshop  Assessments 

4. 1 .5. 1 . Process  Assessments 

Graphs  summarizing  the  responses  on  the  Resiliency  Process  Questionnaire  can  be  found 
in  Appendix  E.  Overall,  participants  in  the  workshop  had  positive  comments  about  the  DC 
process.  Activities  that  participants  liked  the  most  were  the  break-out  sessions  that  focused 
on  the  diverge/converge  activities.  Interestingly,  the  activities  that  participants  valued  the 
least  were  the  team  building  exercises  (Questions  9,  12,  and  30).  In  terms  of  the 
divergence  of  the  group  (Question  18)  no  one  thought  there  were  people  in  the  group  with 
specializations  very  similar  to  their  own.  Finally,  participants  were  satisfied  with  their 
experience  and  would  overwhelmingly  participate  in  similar  projects  in  the  future  (Questions 
26,  27,  and  31). 


4.1 .5.2.  Product  Assessments 

Graphs  summarizing  the  responses  on  the  Resiliency  Product  Questionnaire  can  be  found 
in  Appendix  F.  Areas  where  the  problem  owners  and  the  Executive  Team  differed  on 
responses  addressed  the  uniqueness  of  the  products  (Question  4)  and  the  ability  to 
generate  the  same  results  without  the  diverse  group  of  participants  (Question  6).  These 
differences  can  be  attributed  to  the  information  in  Item  4  in  the  Product  section,  in  which  the 
problem  owners  thought  that  the  DC  workshop  validated  ideas  they  had  already  come  up 
with  previously. 


4.2.  The  Human  Performance  Augmentation  (HPA)  Workshop 
4.2.1.  HPA  Challenge  Statement 

How  can  we  enhance  or  augment  the  performance  of  human  beings? 


4.2.2.  HPA  Workshop  Participants 

The  Executive  Team  evaluated  over  100  individuals  with  backgrounds  in  approximately  80 
different  career  areas  of  interest.  The  team  ultimately  chose  people  in  the  following  12 
areas  to  attend  the  workshop: 

•  Human  Cycle  Optimization  Specialist 

•  Personal  Trainer 

•  Navy  Seal;  Spy  Trainer 

•  Special  Education  Teacher 

•  Chief  Executive  Officer  of  a  Production  and  Machining  Company 

•  Naturopathic  Doctor 

•  Cardiac  Surgeon 
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•  College  Admissions 

•  University  Cross-Country  Coach 

•  Philharmonic  Orchestra  Conductor 

•  Business  Strategy  Consultant  and  Professor 

Additional  participant  demographic  information  can  be  found  in  Appendix  G. 

Figure  3  shows  participant  information  from  the  DiSC  and  Team  Dimensions  Profiles 
administered  pre-workshop. 


HPA 


Key 

■  D(ominance) 


■  i(nfluence) 


■  S(teadiness) 


■  C(onscientiousness) 


HPA 


Key* 


■  Creator 

■  Advancer 


■  Executor 

■  Refiner 


Figure  3.  DiSC  and  Team  Dimension  Summary  for  HPA  Participants 
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4.2.3.  HPA  Agenda 
Day  1 


Time 

Event 

12:00-12:15 

Arrival  and  Check-In 

12:15-1:15 

Working  Lunch  -  Introduction  of  Participants 

1:15-2:15 

Defining  HPA  (Participants) 

2:15-2:30 

Break 

2:30-3:30 

Connecting  the  Dots 

3:30-4:00 

Overview  of  DC;  HPA  (Executive  Team) 

4:00-4:15 

Break 

4:15-5:00 

Diverge:  Metaphoric/Free  Association 

5:00:5:45 

Report  Out  &  Homework 

5:45-6:30 

Social  hour 

Day  2 


Time 

Event 

8:00-8:30 

Check-in  &  Breakfast 

8:30-8:45 

Previous  Day  Recap 

8:45-9:30 

Zoom/Re-Zoom  Activity 

9:30-9:45 

Break 

9:45-10:45 

Diverge  Activity:  Application  to  the  HUMAN  (3  groups) 

10:45-11:45 

Report  Out 

11:45-12:00 

AFRL  HPA  Lens  Shift  (the  problem/context  within  AFRL) 

12:00-1:00 

Lunch 

1:00-3:00 

Converge  Activity:  Categorization  (1  group) 
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Application  to  Human/AF 

3:00-3:15 

Break 

3:15-4:00 

Admin  (Questionnaire  Completion) 

4:00-5:00 

Closing  Recap 

4.2.4.  HPA  Products 

Appendix  H  contains  the  comprehensive  list  of  ideas  generated  by  the  HPA  group.  Listed 
below  is  a  summary  of  the  most  innovative  or  most  impactful  ideas  on  the  HPA  challenge 
according  to  the  Executive  Team: 

1 .  Research  to  increase  integrated  augmentation  approaches,  amalgamating  both 
traditional  and  non-traditional  methods. 

2.  Explore  the  use  and  effect  of  non-traditional  augmentation  on  performance,  e.g. 
auras,  acupuncture,  etc. 

3.  Vary  environmental  factors,  including  tonal  changes  to  maintain  alertness  as  well  as 
the  effects  of  light  on  performance. 

4.  Implement  intentional  distractions  to  keep  people  focused  and  alert . 

4.2.5.  HPA  Workshop  Assessments 

4.2.5. 1 .  Process  Assessments 

Graphs  summarizing  the  responses  on  the  HPA  Process  Questionnaire  can  be  found  in 
Appendix  I.  Overall,  participants  overwhelmingly  liked  the  trust  building  activity  called 
Zoom/Re-Zoom  and  the  diverge  activity  that  focused  on  metaphoric  thinking  and  free 
association  (Question  30).  Also,  about  70%  of  the  participants  felt  that  the  group  was 
diverse  in  terms  of  their  area  of  specialization  (Question  18).  Finally,  participants  were 
satisfied  with  their  experience  and  would  participate  in  similar  projects  in  the  future 
(Questions  26,  27,  and  31). 


4. 2. 5. 2.  Product  Assessments 

Graphs  summarizing  the  responses  on  the  HPA  Product  Questionnaire  can  be  found  in 
Appendix  J.  Again,  the  problem  owners  were  not  convinced  that  they  would  not  have 
achieved  the  same  or  similar  results  without  the  DC  workshop  (Question  6),  however,  they 
did  agree  that  new  research  ideas  were  generated  from  the  workshop  (Question  3)  and  that 
there  was  value  in  the  divergent  part  of  the  process  (Question  7).  Also,  their  overall 
impression  of  the  workshop  was  positive  (Question  14). 
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4.3.  The  Autonomy  Workshop 
4.3.1.  Autonomy  Challenge  Statement 

How  can  machines  seamlessly  integrate  with  humans  to  maximize  performance  in  highly 
complex  and/or  high-risk  environments?  Examples  include  systems  that  require  human 
supervision  of  automated  operations,  such  as  robotic  manufacturing,  remotely  piloted 
aircraft,  and  automated  flight  deck  operations. 


4.3.2.  Autonomy  Workshop  Participants 

After  considering  over  90  individuals  from  more  than  50  potential  career  areas  for  the 
Autonomy  Workshop,  the  Executive  Team  ultimately  selected  13  ideal  candidates  to  attend 
the  event.  The  backgrounds  of  the  chosen  participants  include: 

•  Investment  Manager  and  Certified  Financial  Accountant 

•  Warfighter 

•  Robotic  Surgeon 

•  Peace  Corps  Member 

•  Astronaut 

•  Owner  of  Heavy  Construction  Company 

•  Expert  in  Human-System  Interfaces 

•  Pilot 

•  Remotely  Piloted  Aircraft  Operator 

•  Science,  Technology,  Engineering,  and  Mathematics  Educator;  Instructional 
Designer 

•  Sensors  and  Cognitive  Systems  Researcher 

•  Entrepreneur 

•  Generational  Expert  and  Strategist 

Additional  participant  demographic  information  can  be  found  in  Appendix  K. 

Figure  4  shows  participant  information  from  the  DiSC  and  Team  Dimensions  Profiles 
administered  pre-workshop. 
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Autonomy 


Key 

■  D(ominance) 

■  i(nfluence) 

■  S(teadiness) 

■  C(onscientiousness) 


Autonomy 


Key* 

■  Creator 

■  Advancer 

■  Executor 

■  Refiner 


Figure  4.  DiSC  and  Team  Dimension  Summary  for  Autonomy  Participants 


4.3.3.  Autonomy  Agenda 
Day  1 


Time 

Event 

8:00-8:30 

Arrival  and  Check-In 

8:30-9:30 

Introduction  of  Participants 

9:30-10:30 

Defining  Human-Machine  Teaming  in  Autonomous 
Systems  (Participants) 

10:30-10:45 

Break 

10:45-11:45 

Connecting  the  Dots 

11:45-12:45 

Lunch  &  DC  Overview  (Executive  Team) 

12:45-1:30 

Zoom/Re-zoom  Activity 
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1:30-2:35 

Diverge  Activity:  Break-Out  Session 

2:35-2:50 

Break 

2:50-3:20 

Diverge  Activity:  Break-Out  Session  Continued 

3:30-4:30 

Report  Out  and  Discussion 

4:30-5:30 

AFRL  Autonomy  Lens  Shift  (the  Problem/Context 
within  AFRL) 

5:30-6:30 

Social  hour 

Day  2 


Time 

Event 

8:00-8:30 

Check-In  &  Breakfast 

8:30-9:30 

Previous  Day  Recap  and  Homework  Review 

9:30-10:10 

Converge  Activity  Part  1 

10:10-10:50 

Converge  Activity  Part  1:  Report  out 

10:50-11:05 

Break 

11:05-12:05 

Converge  Activity  Part  II:  Research  themes 

12:05-12:20 

Admin  (Guestionnaire  Completion) 

12:20-12:30 

Closing  Recap 

4.3.4.  Autonomy  Products 

Appendix  L  contains  the  comprehensive  list  of  ideas  generated  by  the  Autonomy  group. 
Listed  below  is  a  summary  of  the  most  innovative  or  most  impactful  ideas  on  the  Autonomy 
challenge  according  to  the  Executive  Team: 

1 .  Research  generational  differences.  In  particular,  how  different  generations  accept 
and  utilize  technology  changes  and  greater  automation,  and  tailor  training  and 
implementation  of  increased  autonomy  so  that  it  takes  into  account  these 
considerations. 

2.  Utilize  a  cafeteria-style  use  structure  of  autonomy.  That  is,  enable  users  to  pick  and 
choose  when  and  where  to  use  autonomy  as  well  as  the  level  of  automation. 
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3.  Managing  autonomy  doing  things  beyond  human  control.  Implement  greater 
automation  to  extend  normal  human  capabilities  or  to  perform  beyond  normal 
capabilities. 

4.3.5.  Autonomy  Workshop  Assessments 

4.3.5. 1 .  Process  Assessments 

Graphs  summarizing  the  responses  on  the  Autonomy  Process  Questionnaire  can  be  found 
in  Appendix  M.  Unlike  the  previous  two  workshops,  participants  in  the  Autonomy  Workshop 
had  varying  opinions  on  many  of  the  questions.  For  instance,  some  people  did  not  think 
that  everyone  participated  in  the  group  discussions  (Question  4)  and  felt  constrained  to  only 
pursue  certain  types  of  ideas  (Question  5).  Most  people  thought  the  level  of  structure  of  the 
workshop  was  just  right,  however,  some  thought  it  was  too  little  and  even  more  thought  it 
was  too  much  (Question  14).  Another  area  of  disagreement  was  whether  the  facilitators 
answered  questions  effectively  during  the  workshop  (Question  17).  Also,  many  participants 
(-45%)  thought  that  the  knowledge  and  experience  of  the  group  was  insufficient  to  explore 
the  problem  (Question  20).  Finally,  this  was  the  first  time  that  any  participants  were 
dissatisfied  with  their  experience  (Question  26)  but  all  agreed  that  they  would  like  to 
participate  in  similar  projects  in  the  future  (Question  27). 


4. 3. 5. 2.  Product  Assessments 

Graphs  summarizing  the  responses  on  the  Autonomy  Product  Questionnaire  can  be  found 
in  Appendix  N.  In  summary,  the  problem  owners  did  not  think  any  new  research  ideas  or 
areas  of  investment  were  uncovered  from  the  workshop  (Question  3  and  5).  The  problem 
owners  thought  that  perhaps  this  occurred  because  the  topic  was  too  broad  (Question  14). 
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5.  DISCUSSION 


This  section  contains  a  discussion  of  the  DC  workshops  in  terms  of  the  process,  products, 
and  research. 


5.1.  DC  Process 


5.1 .1 .  Workshop  Timing 

In  terms  of  the  length  of  the  workshops,  1  and  V2  days  seemed  to  be  optimal.  It  was  a 
reasonable  amount  of  time  for  the  participants  to  commit  to  this  activity  and  an  adequate 
amount  of  time  to  conduct  the  workshop  activities  effectively.  For  the  first  two  workshops,  a 
!4  day  session  was  followed  by  a  full  day.  For  the  third  workshop,  a  full  day  was  followed  by 
a  V2  day  session.  Based  on  Executive  Team  feedback,  there  was  a  clear  advantage  of 
having  the  V2  day  session  first,  followed  by  a  full  day  session. 

For  the  Resiliency  and  HPA  Workshops,  the  first  V2  day  included  team/trust  building, 
introducing  the  problem  in  general  terms,  and  then  engaging  in  the  first  diverge  activity. 
Since  diverging  is  sometimes  uncomfortable  for  people,  it  seemed  reasonable  to  conduct 
one  diverge  activity  on  the  first  day  and  then  finish  up  with  a  social  hour.  On  the  second 
(full)  day  for  the  Resiliency  Workshop,  another  diverge  activity  was  conducted  before 
converging  and  then  wrapping  up.  The  second  (full)  day  for  the  HPA  Workshop  was  a  little 
different  because  of  the  feedback  that  the  Executive  Team  received  from  participants  in  the 
Resiliency  Workshop.  During  the  Resiliency  Workshop,  the  Executive  Team  tried  very  hard 
to  keep  the  discussion  on  resiliency  in  general  terms  and  not  talk  about  the  Air  Force 
specific  resiliency  challenge  because  they  thought  it  would  bound  the  discussions  and 
inhibit  the  “out-of-the-box”  thinking  they  wanted  participants  to  engage  in.  They  wanted 
people  to  talk  openly  about  their  unique  perspectives  on  the  general  problem.  When 
specific  examples  were  brought  up  (i.e.,  the  suicide  rate  of  deployed  military  members)  the 
Executive  Team  tried  to  bring  the  discussion  back  to  a  more  general  discussion.  This, 
however,  caused  many  participants  angst  because  they  knew  they  were  brought  together 
by  the  Air  Force  to  “help”  and  they  wanted  to  ensure  their  inputs  were  relevant  to  the 
specific  Air  Force  problem  that  brought  them  together.  As  a  result,  the  Executive  Team 
decided  that  they  should  present  the  focus  area  of  the  subsequent  workshops  in  an  Air 
Force  context  sometime  later  in  the  workshop  so  as  not  to  constrain  initial  discussions  but  to 
provide  that  context  for  the  participants.  The  Executive  Team  added  a  session  in  the  HPA 
and  Autonomy  Workshops  to  address  this  and  called  it  the  AFRL  lens  shift.  During  the  HPA 
Workshop,  the  second  day  started  with  a  team  building  exercise  followed  by  another 
diverge  activity  and  then  the  lens  shift  before  lunch.  After  lunch,  the  group  participated  in  a 
converge  activity  and  then  wrapped  up.  This  order  of  events  was  very  effective. 

For  the  Autonomy  Workshop,  due  to  scheduling  conflicts  with  the  Executive  Team  and 
participants,  the  first  day  of  the  workshop  was  a  full  day  followed  by  a  V2  day  on  the  second 
day.  Because  of  this,  the  first  day  contained  team  building  activities,  and  two  diverge 
activities  followed  by  the  lens  shift  at  the  end  of  the  day.  Unfortunately,  this  sequence  of 
events  or  the  timing  of  these  events  impacted  the  second  day  significantly.  Perhaps 
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because  participants  had  spent  most  of  the  day  in  the  uncomfortable  area  of  exploring  the 
problem  space  without  moving  to  solutions  and  then  they  were  presented  with  a  very 
specific  problem  during  the  lens  shift,  they  wanted  to  focus  on  problem  solving  the  next  day. 
Perhaps  because  they  had  the  evening  to  think  about  the  day’s  activities  and  the  last  thing 
they  heard  was  the  lens  shift,  they  became  focused  on  solving  the  specific  problem 
presented  in  the  lens  shift  for  the  next  day.  Whatever  the  cause,  the  participants  could  not 
focus  on  anything  but  trying  to  solve  the  Air  Force  specific  problem  presented  at  the  end  of 
Day  1 .  The  Executive  Team  spent  more  time  than  usual  encouraging  the  participants  to 
engage  in  the  prescribed  activities  and  explaining  the  process  in  different  ways  to  help  them 
better  understand  the  objectives  of  the  process.  Even  after  ad  hoc  discussions  on  the 
purpose  of  the  DC  process,  the  majority  of  participants  were  disinclined  to  participate  in 
anything  other  than  problem  solving.  While  the  enthusiasm  and  energy  they  exhibited  in 
focusing  on  solutions  was  admirable,  the  workshop  was  not  designed  to  solve  the  problem. 
This  circumstance  somewhat  derailed  the  second  day  of  the  Autonomy  Workshop. 

5.1.2.  Network  of  Participants 

Another  process  challenge  was  identifying  participants  in  adjacent  fields  to  the  workshop 
focus  area.  As  was  mentioned  in  Section  3.1.1  participants  were  selected  via  personal 
connections,  professional  networks  and  associations,  or  internet  searches.  The  advantage 
of  the  first  two  methods  was  that  knowing  the  participants  gave  the  Executive  Team  a  better 
feel  for  how  well  the  person  would  thrive  in  this  type  of  workshop  and  how  comfortable  the 
person  might  be  staying  in  the  problem  space.  As  a  matter  of  fact,  the  Executive  Team 
determined  that  this  factor  was  most  important  for  the  success  of  the  workshops.  So  if  the 
potential  participants  are  unknown,  it  might  be  risky  to  recruit  them.  Even  the  pre-workshop 
assessments  did  not  give  a  good  indication  of  how  effective  people  would  be  in  this 
capacity.  But  if  selecting  participants  depends  on  relying  only  on  people  the  Executive 
Team  knows  personally,  this  narrows  the  participant  pool  significantly. 

Another  point  to  consider  when  selecting  participants  is  how  diverse  to  make  the  group.  As 
was  mentioned,  people  were  selected  in  fields  adjacent  to  the  area  of  interest  so  the  level  of 
“diversity”  was  somewhat  constrained  by  the  Executive  Team’s  process  for  selecting 
participants.  However,  for  an  initial  trial  of  the  process,  this  seemed  like  a  reasonable 
approach. 

To  help  mitigate  these  challenges  for  future  DC  workshops,  the  Executive  Team  would  like 
to  establish  a  network  of  participants  by  gathering  contacts  over  time  from  previous 
workshop  participants,  WBI  employees  as  well  as  AFRL  contacts. 

5.1.3.  Problem  Owners 

A  third  process  discussion  point  is  the  role  of  the  problem  owner.  As  was  mentioned  earlier, 
the  RH  Director  selected  three  areas  to  investigate  with  the  DC  process.  He  also  selected 
problem  owners  to  serve  as  liaisons  between  the  Executive  Team  and  researchers  at  AFRL, 
working  in  the  areas  of  Resiliency,  HPA,  and  Autonomy.  A  potentially  better  scenario  would 
have  been  if  the  problem  owners  initiated  the  interaction  with  the  IDEA  Lab  and  proactively 
sought  out  the  use  of  the  DC  process  to  help  them  with  their  research  challenge.  That’s  not 
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to  say  that  the  problem  owners  that  were  identified  for  these  three  workshops  did  not 
engage  in  any  of  the  planning  activities;  however,  since  their  involvement  was  directed,  it 
was  minimal.  If,  in  the  future,  problem  owners  are  the  catalyst  for  a  DC  workshop,  their 
level  of  involvement  would  certainly  be  higher  as  would  the  chance  for  products  to 
transition. 

5.2.  DC  Products 

All  three  DC  workshops  generated  products.  The  top  ones  for  each  workshop  are  restated 
here: 

o  Resiliency  Workshop 

•  Continuity  of  care 

•  Institute  peer  mentors 

•  Take  a  holistic  approach  to  care 

•  Validation  of  the  problem  owner’s  conclusions 

o  Human  Performance  Augmentation  Workshop 

•  Integrated  augmentation  approaches  (traditional  and  non-traditional) 

•  Non-traditional  (auras,  acupuncture,  etc.) 

•  Vary  environmental  factors 

•  Intentional  distractions 

o  Autonomy  Workshop 

•  Generational  differences 

•  Cafeteria-style  usage  structure 

•  Managing  autonomy  doing  things  beyond  human  control 

Although  the  Autonomy  Workshop  got  a  little  off  track,  the  ideas  listed  directly  above  are  the 
products  from  the  workshop.  Of  particular  interest  was  the  idea  of  investigating  the  effects 
of  multiple  generations  on  the  design,  development,  and  deployment  of  autonomous 
systems.  For  instance,  people  in  the  Millennial  Generation  will  be  the  future  operators  of 
heavily  automated  systems  and  will  probably  be  very  comfortable  doing  so,  but  the 
researchers  determining  the  optimum  level  of  automation  between  the  human  and  the 
system  are  from  Generation  X  and  they  have  a  different  perspective  on  autonomous 
systems.  Further,  the  generals  in  charge  of  allowing  that  type  of  system  to  operate  in  a 
future  combat  environment  are  from  the  Baby  Boomer  Generation  with  yet  a  third 
perspective  on  the  employment  of  autonomous  systems.  Considering  each  generation  and 
their  approaches  and  perspectives  on  interacting  with  automation  may  play  a  significant  role 
in  effectively  integrating  and  employing  this  type  of  technology  in  the  future.  This  unique 
perspective  on  the  generational  challenges  of  designing  and  employing  an  autonomous 
system  was  brought  to  light  because  one  of  the  participants  of  the  Autonomy  Workshop  was 
a  Generational  Expert  and  Strategist.  Many  of  the  other  participants  acknowledged  that  they 
were  unaware  of  this  factor  when  considering  the  autonomy  challenge.  This  idea  for 
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research,  as  well  as  all  others,  are  now  in  the  hands  of  the  problem  owners  to  pursue  them 
as  they  desire. 

5.3.  DC  Research 

The  members  of  the  Executive  Team  from  RH  were  interested  in  seeing  how  the 
personalities  of  the  participants  as  well  as  their  interaction  style  might  impact  the  process 
and  products  of  the  workshop.  The  pie  charts  in  Sections  4.1.2,  4.2.2,  and  4.3.2  show  the 
distribution  of  participants  in  all  three  workshops  based  on  the  two  assessments 
administered  prior  to  the  workshops  (the  DiSC  and  Team  Dimension  assessments).  In 
terms  of  the  Resiliency  Workshop  participants,  they  were  fairly  evenly  divided  among  three 
of  the  four  DiSC  dimensions  [there  were  zero  participants  in  the  fourth  dimension]  and  all 
four  of  the  four  Team  Dimensions.  For  the  HPA  Workshop  participants,  the  group  was  fairly 
evenly  divided  among  the  four  DiSC  dimensions  and  the  four  Team  Dimensions.  However, 
for  the  Autonomy  Workshop  participants,  the  majority  of  participants  fell  into  the  Dominance 
category  of  DiSC  and  the  Creator  category  for  the  Team  Dimension.  Having  this  imbalance 
may  have  been  the  cause  of  some  of  the  challenges  that  the  Executive  Team  experienced 
when  conducting  the  Autonomy  Workshop. 

Since  the  characteristics  of  the  participants  may  have  had  an  effect  on  the  workshops, 
perhaps  these  assessments  can  be  used  to  screen  participants  in  the  future  or  to,  at  a 
minimum,  balance  break-out  groups  to  enhance  the  interaction  of  the  participants. 
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6.  CONCLUSIONS 


The  value  of  bringing  together  groups  of  people  with  diverse  backgrounds  and  experiences 
to  examine  problems  and  solutions  in  an  innovative  way  was  demonstrated  with  the 
development  and  implementation  of  the  Divergent  Collaboration811  process.  Although  there 
were  challenges  in  developing  the  process  and  executing  the  workshops,  many  interesting 
ideas  were  generated  and  the  lessons  learned  by  participating  in  and  evaluating  the 
feedback  will  be  incorporated  to  make  the  process  even  more  effective  for  future  DC 
workshops. 
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7.  APPENDICES 

7.1.  Appendix  A  -  Executive  Team  Biographies 

Bart  Barthelemy  is  the  Founding  Director  of  the  Wright  Brothers  Institute  (WBI)  and  the 
current  Director  of  WBI’s  IDEA  Lab,  where  he  is  responsible  for  providing  innovative 
solutions  to  critical  problems  in  science,  technology  and  R&D.  He  is  a  nationally  known 
consultant,  facilitator  and  trainer  and  specializes  in  leadership,  as  well  as  innovation  and 
collaboration  for  academia,  government,  and  Fortune  500  companies.  Dr.  Barthelemy  has  a 
Bachelor  of  Science  in  Chemical  Engineering  and  Polymer  Science  from  MIT;  a  Master  of 
Science  in  Nuclear  Engineering  and  Physics  from  MIT,  and  a  Doctor  of  Philosophy  in 
Nuclear  Physics/Mechanical  Engineering  from  The  Ohio  State  University. 

Candace  Treasure  is  the  Innovation  Program  Manager  at  the  Wright  Brothers  Institute’s 
IDEA  Lab.  Utilizing  leading-edge  innovation  techniques  and  tools,  Candace  plans  and 
executes  major  collaborative  innovation  programs  to  help  solve  complex  challenges  in  the 
defense,  science  and  technology,  and  R&D  industries.  She  holds  a  Master’s  degree  in 
Business  Administration  with  a  Bachelor’s  in  International  Business  and  Spanish,  both  from 
the  University  of  Dayton. 

Scott  Galster,  PhD,  is  the  Chief  of  the  Applied  Adaptive  Aiding  Section  in  the  Applied 
Neuroscience  Branch,  711  Human  Performance  Wing  at  Wright-Patterson  Air  Force  Base. 
Scott  leads  the  Applied  Neuroscience  sub-Core  Technology  Competency  area  and  is 
accountable  for  strategic  planning,  development,  execution  and  reporting  of  a  diverse  body 
of  neuroscience-based  research  and  development  efforts.  Scott  holds  a  PhD  in  Applied 
Experimental  Psychology. 

Kristen  Liggett  is  a  Senior  Human  Factors  Engineer  in  the  Battlespace  Visualization  Branch 
of  the  Human  Effectiveness  Directorate  where  she  leads  a  multi-disciplinary  team  of 
researchers  addressing  the  challenging  area  of  information  overload  in  cyber  operations 
and  how  visualizations  can  help  alleviate  this  situation.  Kristen  received  her  BS  in  Human 
Factors  Engineering  in  1989,  a  MS  in  Engineering  Management  in  1994,  and  a  PhD  in 
Engineering  in  2000.  Through  her  26  years  working  for  the  government,  Kristen  has  seen 
the  value  of  bringing  a  diverse  group  of  people  together  to  solve  complex  problems  and  has 
been  involved  with  Divergent  Collaboration311  since  its  inception. 

2d  Lt  Rachael  Bryson  is  in  the  Applied  Neuroscience  Branch  of  the  71 1  Human 
Performance  Wing  at  Wright-Patterson  Air  Force  Base.  Rachael  leads  the  Applied 
Neuroscience  Cyber  Research  and  Development  effort.  She  is  responsible  for  strategic 
planning,  development,  execution  and  reporting  of  a  diverse  body  of  research  and 
development  efforts  examining  the  applied  neuroscience  aspects  of  decision  making  in 
cyber,  ISR,  and  RPA  domains.  She  has  a  Bachelor  of  Arts  in  Psychology  and  Studio  Art  and 
a  minor  in  Middle  Eastern  Migrations  and  Communities,  all  from  James  Madison  University. 

Kathy  Hollingsworth  is  dedicated  to  helping  others  reach  their  potential  by  understanding 
their  worth  using  iiChange®  skills  and  tools  to  enrich  their  lives.  Prior  to  starting  Innovative 
Interchange®  Inc.,  she  spent  twenty-nine  years  in  banking,  including  several  years  as 
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president  and  CEO  of  National  City  Bank,  Southwest.  She  earned  a  bachelor’s  degree  in 
geography  from  Wittenberg  University  and  an  MBA  from  Case  Western  University. 
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7.2.  Appendix  B  -  Invitation  to  Participants 

Dear  [ name  of  inviteel: 

The  IDEA  Lab  at  the  Wright  Brothers  Institute,  Dayton,  OH,  in  conjunction  with  a  team  from 
the  Air  Force  Research  Laboratory  (AFRL)  at  Wright-Patterson  Air  Force  Base,  has  created 
a  unique  approach  to  help  achieve  breakthroughs  in  solving  complex  problems.  This 
process,  known  as  Divergent  Collaboration5",  takes  place  during  a  two-day  workshop 
designed  to  stimulate  innovation  by  bringing  together  a  diverse  group  of  people  to  explore  a 
specific  technology  challenge  in  a  new  way. 

Divergent  Collaboration5"  emerged  as  a  result  of  a  Tech-Arts  Project  that  brought  together 
local  artists  and  engineers  in  January  201 1 .  This  pilot  project  was  successful  in  infusing  new 
ideas  and  non-traditional  thinking  into  AFRL’s  information  visualization  technology 
development  activities. 

While  most  of  us  collaborate  with  people  we  know  or  who  have  similar  backgrounds,  this 
static  approach  to  problem-solving  often  produces  predictable  results.  That’s  where  you 
come  in.  Based  on  your  area  of  expertise  and  the  unique  perspective  you  can  provide  as  a 
[profession/ area  of  expertise]  we  would  like  to  invite  you  to  participate  in  a  Divergent 
Collaboration5"  workshop.  [You  were  recommended  by _ /. 

By  harnessing  the  mental  models  and  tools  from  a  variety  of  professions  operating  adjacent 
to  the  problem  at  hand,  we  believe  novel  and  innovative  outcomes  will  result.  Your  distinct 
background  and  knowledge  can  provide  new  insights  into  potential  solutions  to  the 
challenge  facing  the  U.S.  Air  Force  today:  [insert  problem].  \ Brief,  one  sentence  explanation 
of  the  probleml. 

The  workshop  will  take  place  [insert  date 1  at  the  TecAEdge  facility  in  Dayton,  OH.  [If 
participant  is  travelling:  We  are  prepared  to  compensate  your  travel  costs  to  attend.  If 
participant  is  not  AF  or  DoD:  We  are  prepared  to  offer  compensation  based  on  your  time 
and  effort].  In  order  to  measure  the  effectiveness  of  this  workshop,  assessments  dealing 
with  the  process,  outcomes,  and  participants,  in  the  form  of  questionnaires,  will  also  be 
administered.  Lunch  will  be  provided  both  days,  as  well  as  an  evening  social  hour  on  the 
first  day. 

We  appreciate  and  greatly  value  your  participation  in  this  exciting  opportunity  for 
collaboration.  You  can  find  more  information  on  Divergent  Collaboration5",  the  challenge,  the 
IDEA  Lab,  and  the  team  leading  this  effort,  in  the  attached  flyer.  Please  respond  with  your 
participation  availability  as  soon  as  possible  but  no  later  than  [insert  date 1  via  email 

at  candace.treasure@wbi-icc.com. 

Sincerely, 

Candace  Treasure  and  the  Divergent  Collaboration5"  team 
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Gender 


•  Female: 

•  Male 


Age: 


•  20-29 

•  30-39 

•  40-49 

•  50-59 

•  Over  60 


Education: 


6  (43%) 
8  (57%) 


3  (21%) 

1  (7%) 

2  (14%) 

3  (21%) 

5  (36%) 


Post  Master’s  Degree 

8 

(57%) 

Master’s  Degree 

3 

(21%) 

Undergraduate  Degree 

2 

(14%) 

High  School  Diploma 

1 

(7%) 
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7.4.  Appendix  D  -  Comprehensive  List  of  Ideas  from  the  Resiliency  Grou 

•  Provide  emotional  support 

-  Mentor;  ongoing  and  consistent  relationships;  enhance  supervisor  training; 
change  stigma  of  getting  help;  better  assessment  tools 

•  Improve  work  environment 

-  Human  factors;  sensory  integration;  enhance  sense  of  community  through 
technology  and  social  media 

•  Required  group  debriefs 

•  Physical  activity  on  a  regular  basis 

•  Awareness  of  effects  (kills) 

•  Holistic  approach  of  looking  at  warfighter’s  life 

•  Awareness  of  resiliency  -  training,  best  practices,  how  to  be  resilient 

•  Options  for  open  and  anonymous  feedback 

•  Longitudinal  studies  of  warfighters 

•  Required  single  status 

•  Controlled  social  support 

•  “A  day  in  the  life” 

-  Simulation  for  spouses,  girlfriends,  family,  or  support  groups  to  see  the 
work  their  loved  one  does  on  a  daily  basis 

•  Educate  the  public 

•  Individual  assessments 

-  EEG/FMRI;  social  networks;  expectation  checking;  perception  checking 

•  Continuity  of  Care 

-  Pre,  during,  and  post  assignment 

•  Institute  selection  requirements  in  terms  of  resiliency  of  individuals 

•  Identify  variables  that  make  a  person  resilient  and  support  them  in  continuing  that 
behavior 

•  Explore  Forensics 

•  Morale  assessment 

•  Educate  people  on  how  to  move  in  and  out  of  structure  all  the  time 

•  Institute  teacher/mentor,  preferably  a  peer 

•  Generalize  physical  mission  training  to  individual  wellness 

•  Technological  advances  that  impact  morality 
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•  Leadership  supports  life  resiliency  activities  and  gets  graded  on  it. 

•  Provide  education/services  for  people  who  want  to  take  advantage  of  them  and 
incentive  program  participation 
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7.5.  Appendix  E  -  Summary  of  Resiliency  Process  Questionnaires 


Question  1 :  I  came  to  the  workshop  with  clear  expectations. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Comments  from  particpants: 


•  “I  mostly  disagree,  but  more  clarity  would  have  been  problematic/limiting 
creativity” 

•  “Tried  to  be  open  minded” 

•  “A  bit  ‘fuzzy’  but  this  actually  made  the  workshop  more  spontaneous  and  perhaps 
more  authentic” 


Question  2:  The  pre-planning  was: 


100% 

90% 

80% 

70% 

60% 

50% 

40% 
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■  I  I  ■  I  I  ■  I  t  ■ 


Completely  Mostly  Agree  Neither 
Agree  Agree  nor 

Disagree 

Comments  from  participants: 


Mostly  Completely 

Disagree  Disagree 


■  Well  Articulated 

■  Helpful 

■  Detailed  Enough 
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•  “Good  that  not  much  info  provided  at  first” 

Question  3:  Terms  of  engagement  were  clear^ 

Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Question  4:  Everyone  participated  in  the  group  discussion  during  the  workshop. 

Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  5:  I  felt  constrained  to  only  pursue  certain  types  of  ideas. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  6:  The  discussions  during  the  workshop  were  valuable. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  7:  The  process  and  activities  used  in  this  workshop  were  essential  in 
helping  our  group  meet  its  objective. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  8:  The  Process  Diagram  helped  me  to  track  progress  during  the  workshop. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 
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Question  9:  Rate  the  activities  below  in  terms  of  if  you  think  they  were  valuable  to 
the  workshop. 
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■  Extremely  Worthless 

■  Somewhat  Worthless 

■  Neutral 

■  Somewhat  Valuable 

■  Extremely  Valuable 


Question  10:  The  process  and  activities  used  in  the  workshop  significantly  enhanced 
the  creativity  of  the  group’s  ideas. 

Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 
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Question  11:  The  overall  process  flowed  well. 


Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  12:  Rate  the  length  of  time  spent  in  each  of  the  following  areas  during  the 
workshop. 
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Question  13:  The  group  took  full  advantage  of  the  activities  introduced  during  the 
workshop. 


Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90% 


Question  14:  Rate  the  level  of  “structure”  for  this  workshop. 


Just  right 
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Question  15:  Overall,  the  workshop  facilitators  were  effective. 


Comments  from  participants: 

•  "On  target-listening-helpful-never  too  heavy  loaded  but  kept  us  on  course" 

•  “Some  people  tried  to  sell  ideas  (products)  too  much” 

Question  16:  The  workshop  facilitators  had  sufficient  knowledge  to  instruct  our 
group  on  the  process  used  during  the  workshop^ _ 

Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  17:  The  workshop  facilitators  answered  questions  effectively  during  the 
workshop. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 

Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Comments  from  participants: 

•  “Very  helpful  to  have  others  (facilitators  and/or  project  people)  create  group  flip 
charts  so  we  stayed  focused  on  topics  and  ideas  flowing” 

Question  18:  Describe  your  area  of  specialization  as  compared  to  the  other  members 
of  the  workshop. _ 


Very  Similar  to  Mine 


Somewhat  Similar  to  Mine 


Somewhat  Different  from  Mine 


Very  Different  from  Mine 


Other  (Please  Explain) 


B 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Comments  from  participants: 

•  "Different  specialties  but  similar  creative  and  free  thinking" 
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Question  19:  I  had  a  great  deal  of  prior  knowledge/expertise  on  the  topic/problem  we 
examined  in  this  workshop. _ 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


- 

■- 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  20:  The  knowledge  and  experience  of  the  workshop  participants  were 
sufficient  to  explore  this  problem.  _ 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Comments  from  participants: 

•  "Should  have  people  who  were  deployed  in  units/activities  in  question" 
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Question  21:  The  ideas  generated  during  the  workshop  were  valuable. 


Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  22:  The  workshop  gave  me  the  opportunity  to  express  my  ideas^ 

Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  23:  At  one  or  more  points  during  the  workshop,  the  group  approached  the 
problem  in  a  unique  way. 


Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Comments  from  participants: 

•  "Peace  and  war  as  resilient  ideas  captured  so  many  ways  people  are  resilient.” 

Question  24:  In  looking  back,  I  believe  there  were  ideas  that  our  group 
overlooked/dismissed. 

Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Comments  from  participants: 

•  "Could  have  used  a  bit  more  problem  background  to  trigger  ideas.” 

•  "Nothing  major  but  of  course  there  are  many  ways  of  thinking . .  .the  universe  of 
possibilities  is  infinite.” 
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Question  25:  We  accomplished  our  primary  purpose. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  26:  How  satisfied  were  you  with  your  experience? 
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Question  27:  I  would  like  to  participate  in  similar  projects  in  the  future. 


Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  28:  In  future  workshops,  I  would  recommend  (check  all  that  apply 

50% 

45% 

40% 

35% 

30% 

25% 


Comments  from  participants: 

•  "I  liked  the  way  it  was  conducted” 

•  “No,  comes  out  in  the  group”  (in  regards  to  "better  definition  of  problem”) 

•  “More  break-outs... and  serious  group  homework,  if  possible” 

•  “better  definition  of  what  has  been  done  by  the  team  from  WP” 

•  "Less  large  group  time  during  ‘why’  am  I  here  part” 

•  "More  participants  with  direct  experience.  Maybe  some  unsuccessful  too” 
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Question  29:  Did  the  location  of  the  workshop  meet  your  needs? 


Comments  from  participants: 
•  "Very  good" 


Question  30:  What  activities  did  you  like  most/least? 

Participant  Comments: 

MOST 

•  “Open  discussion  on  the  areas  of  opportunity  to  solve  the  problem” 

•  “Group  discussion” 

•  “(1 )  Connecting  the  dots  after  we’d  interacted  as  a  group  to  define  resiliency.  (2) 
Build  a  Life  -  how  we  think  -  how  we  work  -  how  we  listen  -  a  great  reminder  for 
being  open  to  everyone.  (3)  Divergent  activity  on  2nd  day  in  AM” 

•  “Breaking  off  in  groups  and  returning  to  large  group  to  converge” 

•  “I  liked  the  converging  the  most” 

•  “I  enjoyed  breakout  sessions  however  they  were  challenging  due  to  the 
“divergent”  personalities  and  styles  in  the  group!” 

•  “Group  discussions  and  zoom  and  re-zoom” 

•  “Break-outs” 

•  “The  zoom  and  re-zoom  I  liked  the  most” 

•  “Defining  problems  from  different  perspectives” 

•  “Converging” 

LEAST 

•  “Too  much  time  (for  me)  spent  on  zoom  and  re-zoom” 

•  “Build  a  life” 

•  “Too  truths.  I  would  have  liked  to  have  a  brief  description  of  who  they  are  and 
what  they  do,  though  it  did  provide  a  good  mind  stretch  activity” 

•  “Two  Lies  and  a  Truth” 

•  “Listening  to  long  stories,  some  not  relevant  to  the  topic” 
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•  “Two  truths  activity” 

•  “I  really  didn’t  have  an  activity  that  I  liked  least” 

•  “Maybe  another  point  of  this  is  what  I  liked  least.  We  kept  returning  and  rehashing 
same/similar  ideas  or  experiences.  Also  maybe  too  abstract  with  nothing  to 
measure  to  show  if  we  could  affect  the  problem” 

POC  Comments: 

•  “Metaphor  video  was  a  good  and  simple  activity.  Separated  groups  were  key! 
Good  job.” 

Question  31:  What  was  your  overall  impression  of  the  Divergent  Collaboration8" 

Workshop? 

Participant  Comments: 

•  “It  was  an  amazing  experience,  variety  of  experience  and  opinions  created  many 
different  solutions  of  routes  to  take” 

•  “As  I  told  Candace  yesterday,  I  really  appreciated  being  part  of  a  2  day  think  tank. 
The  workshop  gave  me/us  the  feeling  that  we  were  a  part  of  something  much 
bigger  that  could  change  the  world  -  thank  you” 

•  “I  found  both  the  conceptual  content  and  the  process  fascinating” 

•  “It  was  a  positive  experience  -  eye  opening  as  to  the  challenges  the  Air  Force  is 
addressing  (size  and  scope).  The  facilitation  team  had  created  a  well  designed 
workshop  that  allowed  and  encouraged  participants  to  be  open,  to  share,  to  trust 
one  another  and  to  see  real  value  in  the  work  we  were  asked  to  do” 

•  “Interesting  and  worthwhile  because  of  well  planned  session  and  group  of 
diverging  individuals” 

•  “(1 )  Excellent,  knowledgeable  and  diverse  participants,  who  appeared  to  fill  in  all 
the  gaps.  (2)  Excellent  guidance  and  stimulation  of  the  participants  in  creative  and 
open  thinking  to  improve  problem  solving  skills.  (3)  I  feel  very  useful  suggestions 
were  recommended  to  the  Divergent  Collaboration811  project.” 

•  “Very  interesting  concept.  The  participant  who  had  experienced  being  deployed 
was  valuable  to  the  group.  Facilitators  were  helpful  and  kept  the  group  on  track” 

•  “Very  good.  Not  sure  if  we  actually  “solved  the  problem”.  I  feel  like  there  might  be 
research/tools  already  developed  that  would  help  (i.e.,  The  Resilience  Scale).” 

•  “Fun,  interesting  approach,  nice  break  from  daily  work,  productive,  real  question 
will  be  translation  into  effective  solutions” 

•  “I  believe  this  workshop  was  very  effective.  Not  only  did  I  learn  a  lot  about  the 
U.S.  Air  Force,  I  feel  like  I  brought  a  lot  of  good  concepts  to  the  table  to  assist  the 
Air  Force  in  being  resilient.” 

•  “Effective  in  learning  varied  approaches  to  a  problem.  Must  now  be  ‘firmed  up’ 
with  definition  of  ways  to  evaluate  success.” 

•  “Great  experience/great  facility.  Glad  to  participate/would  like  to  again. 
Knowledgeable,  friendly  people  who  believe  in  their  mission.” 

POC  Comments: 

•  “Good  ->  from  my  perspective,  it  appears  that  if  this  was  the  first  step  in  this 
whole  process  it  would  have  been  much  more  valuable.” 
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•  “(1 )  The  process  can  be  a  motivational  tool.  (2)  It  is  helpful  to  validate  ideas.  (3) 
Overall,  I  was  satisfied.  However,  I  do  not  think  it  led  to  major  changes  in 
perspective.  More  time  is  needed  to  familiarize  participants  to  the  problem  space. 
For  example,  I  assume  the  art-engineer  project  took  months.” 

Question  32:  What  changes  could  we  make  to  improve  future  projects? 

Participant  Comments: 

•  “More  time” 

•  “I  really  enjoyed  this  process.  I  would,  however,  appreciate  hearing  more  about 
what  you  -  the  experts  -  have  already  found  in  your  research  about  resiliency” 

•  “Be  more  aware  of  those  who  haven’t  shared  and  solicit  their  opinions.  Monitor 
the  amount  of  time  those  who  are  extraverted  take  to  share  their  ideas” 

•  “I  like  the  V  shape  room  but  too  many  people  were  concentrated  at  the  other  end 
of  the  room  -  that  I  felt  separated  from  them.  Loved  having  windows  -  good  food 
-  great  chairs  -  and  a  project  I  highly  value.” 

•  “More  people  who  were  deployed  in  groups  and  situations  in  question.  Perhaps 
broaden  your  participant  base  ->  artists,  electricians/carpenters/retail/advertising” 

•  “I  would  have  liked  to  hear  the  areas  where  majority  of  problems  occurred.  During 
most  of  the  workshop  I  was  picturing  the  combat  type  of  soldier,  not  “maintenance 
crew”.  It  might  be  beneficial  to  involve  a  business  leader  and  also  a  representative 
who  has  close  contact  with  the  typical  “soldier”  or  military  person  such  as  a  school 
teacher  who  deals  with  large  volume  of  teens  who  may  enlist” 

•  “A  more  diverse  group  -  I  think  you  tried,  but  we  ended  up  being  more  similar 
than  different” 

•  “(1 )  It  would  be  good  to  hear  some  outcomes  downstream  (2)  1-2  facilitators  lost 
track  of  rules  of  engagement  by  pushing  back  in  ‘divergence’  with  ‘this  is  what  we 
do’  or  ‘we’ve  tried  that’  limiting  comments.  Not  a  big  problem,  but  did  occur  (3) 
Controlling  off-topic  story  telling  is  always  a  challenge” 

•  “I  don’t  see  any  changes  needed  besides  continuing  to  strive  for  diversity  because 
in  my  experiences  different  opinions/ideas  make  the  problem  solving  much  more 
elaborate  and  considerate  to  all  walks  of  life” 

•  “Someone  made  a  comment  that  still  more  diversity  might  be  useful.  Idea  I  think 
was  that  most  in  this  session  have  been  successful  (resilient)  and  that  somehow 
others  might  broaden  ideas” 

•  “(1 )  Pre-reading,  data  that  described  the  current  state,  etc.  might/would  have 
helped  thought  process  (2)  Explanation  of  ‘current  state’  of  affair  -  i.e.  case  study 
model  would  have  been  helpful  (3)  Now  that  I  am  interested  in  the  subject,  how 
do  I  continue  to  contribute?  Make  this  activity  less  episodic  -  i.e.  bring  the  team 
back  in  1-2-3  months  to  share  further  thoughts  or  discoveries  made  based  upon 
the  exposure  provided  over  the  1st  2  days.  Thoughts  and  ideas  take  time  to 
percolate!” 

POC  Comments: 

•  “Try  to  keep  participants  from  selling  ideas  (products)” 

•  “It  could:  (1 )  be  broken  up  into  smaller  sessions  over  more  days  -  sort  of  “over  it” 
by  end  of  2nd  day,  or  (2)  Shortened  in  general” 
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Question  33:  Comments  and/or  Suggestions. 

Participant  Comments: 

•  “I  just  wanted  to  thank  you  for  the  invitation.  This  was  an  amazing  opportunity, 
one  which  I  have  never  had.  The  concept  of  diverge  ->  converge  is  an  incredible 
tool,  it  is  a  way  of  thinking  that  I  will  try  to  take  home  with  me  and  apply  to  my 
daily  life.  One  thing  that  was  lacking  was  at  least  one  airman  from  the  USAF.  It 
was  good  that  I  was  there  but  since  this  is  for  the  Air  Force  I  feel  that  an  airman 
from  a  Main  Squad  should  have  been  invited/voluntold  to  come  and  contribute.” 

•  “A  thoroughly  enjoyable  day-and-a-half!  My  only  suggestion  would,  again,  be 
monitoring  who  is  participating  and  who  is  not  so  the  process  includes/involves 
every  person” 

•  “Very  professional!  Thank  you  for  this  opportunity.” 

•  “The  toys  at  the  table  were  a  plus” 

•  “Fighter  food  -  high  carb/meat  can  be  sleep  inducing;  mid-day  snacks;  stick  with 
cookies  for  dessert.  Thanks!” 

•  “It  is  a  delicate  balance,  but  more  attention  needs  to  be  paid  to  not  allowing 
people  to  go  off  on  tangents.  As  a  former  consultant,  I  “checked  out”  when  the 
facilitator  allowed  a  participant  to  give  a  20  minute  presentation  on  her 
research/profession.  I  remained  checked-out  during  the  rest  of  the  morning 
activities.  Others  in  the  small  break-out  group  appeared  to  agree,  and  another 
participant  is  the  one  that  actually  brought  us  back.” 

•  “This  was  a  great  2  day  workshop.  The  people  from  all  different  fields  and 
professions  provided  a  rich  insight  on  what  resiliency  is  as  well  as  how  we  believe 
we  can  implement  it  in  the  U.S.  Air  Force.  I  must  say  in  my  own  opinion,  whatever 
changes  are  going  to  be  implemented,  I  would  suggest  a  pilot  study  first  to  ensure 
that  the  culture  of  the  Air  Force  is  not  completely  changed.  As  far  as  assessments 
to  examine  resiliency,  I  hope  that  a  longitudinal  study  is  started  ASAP  to  examine 
the  characteristics  or  traits  that  correlate  with  Resiliency.  Without  that  taking 
assessments  are  pointless.” 

•  “I  actually  enjoyed  most  of  this  -  especially  hearing  how  others  viewed  the  similar 
problem” 

•  “Would  very  much  like  to  get  contact  information  of  all  the  attendees” 

POC  Comments: 

None 
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7.6.  Appendix  F  -  Summary  of  Resiliency  Product  Questionnaires 


Question  1:  The  workshop  helped  me  generate  new  insights  into  the  resiliency 
challenge. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  20%  40%  60%  80%  100% 


l  Exec  Team 

AFRL  Resiliency 
Team 


Comments  from  Exec  Team: 

•  "The  most  profound  thing  Tony  said:  ‘people  have  to  believe  that  our  country  is 
great  and  war  is  necessary  to  help  others.  If  they  don’t  believe  strongly  in  that, 
they  have  problems  over  there.  But  when  they  come  home  and  they  see  all  of  the 
country’s  warts,  they  become  disillusioned  and  all  of  the  reasons  they  used  to 
justify  what  they  did  come  into  questions.  Then  they  question  themselves.  Then 
they  lose  hope.  I  think  that  scenario  can  describe  anyone  who  becomes  suicidal 
-  disillusioned,  questioning  themselves  and  others,  hopeless.’ " 
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Question  2:  At  one  or  more  points  during  the  workshop,  the  problem  space  was 
explored  in  a  unique  way. _ 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


i  Exec  Team 


AFRL  Resiliency 
Team 


0% 


20% 


40% 


60% 


80%  100% 


Comments  from  Exec  Team: 

•  "During  the  metaphor  session,  we  landed  on  the  idea  of  independent  resiliency 
and  dependent  resiliency.  Independent  resiliency  was  attributed  to  bugs, 
animals,  and  plants;  dependent  resiliency  was  attributed  to  things  that  require 
human  maintenance  to  remain  resilient,  such  as  bridges,  buildings,  etc.  We 
discussed  human  resiliency  as  a  combination  of  both.  This  might  be  a  good  way 
to  categorize  different  efforts." 


Question  3:  There  were  new  research  ideas  generated  from  the  workshop. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disgree 


■  Exec  Team 

■  AFRL  Resiliency 
Team 


Completely  Disgree 


0%  20%  40%  60%  80%  100% 


Question  4:  Please  assess  the  uniqueness/creativity  of  the  workshop  outcomes. 
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Outcomes  were  highly 
unique/creative 

Outcomes  were  above  average 
in  uniqueness/creativity 

Outcomes  were  moderately 
unique/creative 

Outcomes  showed  little 
uniqueness/creativity 

Outcomes  were  not 
unique/creative 

N/AThe  group  did  not  propose 
outcomes 


:  Exec  Team 

AFRL  Resiliency 
Team 


0% 


20%  40% 


60% 


80%  100% 


Question  5:  The  workshop  helped  me  uncover  key  areas  for  investment. 


Completely  Agree 


Mostly  Aagree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 
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Question  6:  I  could  have  achieved  the  same  or  similar  results  without  the  Divergent 
Collaboration81  Workshop. 


Question  7:  There  was  value  jn  the  divergent  part  of  the  process. 
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Question  8:  The  divergent  part  of  the  workshop  impacted  the  innovativeness  of  the 
products. 


Completely  Agree 

MostlyAgree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  20%  40%  60%  80%  100% 


■  Exec  Team 


■  AFRL  Resiliency 
Team 


Comments  from  Exec  Team 


•  "Divergent  part  was  important  but  I  don’t  think  “products”  were  an  outcome  of  the 
process.  However  there  was  an  interesting  idea  raised  that  I  thought  was 
innovative:  Can  we  leverage  media  addiction  to  gauge  resiliency?  Is  there  a 
correlation  between  resiliency  and  Facebook  time?" 


Question  9:  The  workshop  helped  me  broaden  my  definition  of  resiliency^ 


Exec  Team 

AFRL  Resiliency 
Team 


Comments  from  Exec  Team: 

•  "Think  of  resiliency  like  the  stock  market  -  sometimes  it’s  up,  sometimes  it’s 
down,  sometimes  it’s  steady-state.  Resiliency  is  a  response  to  change  (or  lack  of 
change).  ‘Influenced  by  perceptions." 
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Question  10:  The  workshop  helped  me  think  about  resiliency  in  a  different  way. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disgree 


Exec  Team 


AFRL  Resiliency 
Team 


0% 


20% 


40% 


60% 


80%  100% 


Question  11:  The  workshop  results  will  impact  my  future  or  existing  research 
activities. 


Completely  Agree 

Mostly  Agree 

■  Exec  Team 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 

K  AFRL  Resiliency 
Team 

0% 

0 

%  2C 

%  4C 

%  60 

%  80 

%  10 
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Question  12:  I  would  use  Divergent  Collaboration9*  again. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


■  Exec  Team 

■  AFRL  Resiliency 
Team 


0%  20%  40%  60%  80%  100% 


Question  13:  I  would  recommend  a  Divergent  Collaboration9*  workshop  to  other 
AFRL  researchers. 


Completely  Agree 

Mostly  Agree 


Neither  Agree  nor  Disagree 

Mostly  Disgree 


Completely  Disagree 


■  Exec  Team 


■  AFRL  Resiliency 
Team 


0%  20%  40%  60%  80%  100% 


Question  14:  What  was  your  overall  impression  of  the  Divergent  Collaboration9* 
Workshop? 

Comments  from  Exec  Team: 

•  "A  great  start" 

•  “I  thought  the  workshop  was  productive.  It  got  people  talking  and  thinking  in  new 
ways.  I  have  two  comments  about  Kevin’s  feedback,  which  was  that  it  served  to 
validate  what  he  already  knew: 

1 .  Validation  from  such  a  diverse  group  of  people  is  a  good  outcome. 
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2.  Even  though  it  validated  the  key  points,  it  gave  him  insight  into  details  that  he 
didn’t  already  have.  For  instance,  he  said  they  already  discussed  having  a 
mentor's  to  support  returning  active  duty  but  he  didn’t  think  about  what  that 
background  and  skills  that  mentor  should  possess.  During  the  workshop  there 
was  a  discussion  about  what  type  of  person  would  be  best  suited  to  mentor  from 
a  War-fighter  standpoint  and  it  was  totally  different  than  what  Kevin  and  Co  were 
thinking. 

3.  Even  though  it  validated  the  key  points,  it  helped  Kevin  reframe  some  ideas 
(continuity  of  Care  concept). 

4.  I  think,  given  his  situation,  he  was  leaning  toward  wanting  validation  and  not 
starting  anything  new  before  he  handed  off  the  program." 

Comments  from  POC: 

•  "I  think  more  time  together  would  be  helpful.  In  fact,  several  meetings  over  a 
couple  weeks." 

•  "Good,  there  are  definitely  benefits.  I  can  see  it  being  shorter  or  shorter  sessions 
over  more  days  (probably  not  logistically  possible).” 


Question  15:  What  changes  could  we  make  to  improve  future  workshops? 

Comments  from  Exec  Team: 

•  "No  ball  throwing  -  randomly  assign  numbers  on  table  tents  and  then  go 
sequentially  down  with  odds,  the  evens,  etc  and  use  different  ways  to  start  the 
sequences  for  each  activity." 

•  “Perhaps  start  with  a  full  day  and  end  with  a  half  day.  That  would  mirror  the 
process — spending  a  day  on  diverging  and  a  half  day  on  converging  (in  general).” 

•  “More  time.  I  felt  like  the  “solution”  phase  was  pretty  rushed.  I  think  we  could 
have  come  up  with  better  ideas  with  increased  amount  of  time.” 

Comments  from  POC: 

•  "Greater  facilitator  control  over  break  outs" 

Question  16:  Comments  and/or  Suggestions 

Comments  from  Exec  Team: 

•  "Stronger  coffee.” 
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7.7.  Appendix  G  -  Demographic  Information  for  HPA  Participants 

Gender 


Age: 


Education: 


Female 

6 

(46%) 

Male 

7 

(54%) 

20-29 

1 

(8%) 

30-39 

4 

(31%) 

40-49 

2 

(15%) 

50-59 

3 

(23%) 

Over  60 

3 

(23%) 

Post  Master’s  Degree 

7 

(54%) 

Master’s  Degree 

2 

(15%) 

Undergraduate  Degree 

2 

(15%) 

High  School  Diploma 

2 

(15%) 
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7.8.  Appendix  H  -  Comprehensive  List  of  Ideas  from  the  HPA  Group 

1 .  Wild  Ideas 

•  Stimulating  brain/body 

•  Acupuncture/acupressure 

•  Measure  electromagnetic  Aura 

•  Use  hypnosis 

•  Measure  Non-traditional  Human  Energies 

•  New  paradigm  for  conops 

•  Train  to  induce  vivid  imagination 

•  Holistic  Approach 

•  Personalize  the  job 

•  System  technology  changes 

•  Prevention 

•  Intentional  Distraction 

•  Openness  to  new  outside  ideas 

•  Use  monitoring  data  to  augment  performance  capability 

2.  Actionable  Research 

•  Spiritual,  social,  cognitive,  extraordinary  (beyond  the  science) 

•  Is  the  human  the  best  performer? 

•  Put  person  in  crisis 

•  Change  contact  for  operation 

•  Make  performance  need  personal 

•  Make  performance  personal 

•  Examine  best  practices  then  apply 

•  Problem  solving  vs.  blaming 

•  Flexibility  in  approach 

•  Rotation 

•  Finding  place  of  most  disruption 

•  Molecular,  biological,  sensing  R&D 

•  Finding  trends 

•  Human  awareness 
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•  Practical  Assessments 

•  Assessing  individual  baseline  for  “work” 

•  Non-intrusive  assessment 

•  Examine  situations  that  lead  to  HPA 

•  Incentive  for  baseline  and  high  performance 

•  Physical  fitness  and  HPA  R&D 

•  Mental  exercises  to  increase  HPA 

•  Individually  derived  incentive 

•  Personal  reward/motivation  approaches  to  HPA 

•  Specific  environment  conditions  to  achieve  HPA  (atm,  audio,  music,  senses) 
3.  Products 

•  Tone  changes  to  maintain  alertness 

•  Stress  Assessment  with  heightened  alertness 

•  Functional  movement  screen 

•  Cycle  measurement  and  application 

•  Daily  objective  and  feedback  tools 

•  Eye,  ear  products 

•  Zero  error  “program” 

•  Screen  moving  with  operator 

•  Team  monitoring  with  feedback 

•  Nutrient  pumps  to  enhance  performance 

•  Automatic  throttling  of  information  base  on  ability 
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7.9.  Appendix  I  -  Summary  of  HPA  Process  Questionnaires 


Question  1:  I  came  to  the  workshop  with  clear  expectations. 


Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 

0% 

0 
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%  30 
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%  70 

%  80 

%  90 

%  10 

Comments  from  participants: 

•  “This  was  on  purpose” 


Question  2:  The  pre-planning  was: 


100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 
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Well  Articulated 
l  Helpful 

Detailed  Enough 


Completely  Mostly  Agree  Neither  Mostly  Completely 
Agree  Agree  nor  Disagree  Disagree 

Disagree 
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Question  3:  Terms  of  engagement  were  clear. 

Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  4:  Everyone  participated  in  the  group  discussion  during  the  workshop. 

Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Distribution  A:  Approved  for  public  release;  distribution  unlimited. 
88  ABW  Cleared  06/18/2013;  88ABW-20 13-2894. 

60 


Question  5:  I  felt  constrained  to  only  pursue  certain  types  ofjdeas. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  6:  The  discussions  during  the  workshop  were  valuable. 

Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  7:  The  process  and  activities  used  in  this  workshop  were  essential  in 
helping  our  group  meet  its  objective. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  8:  The  Process  Diagram  helped  me  to  track  progress  during  the  workshop. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  9:  Rate  the  activities  below  in  terms  of  if  you  think  they  were  valuable  to 
the  workshop. 


100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 


1 

1 

_ 

1 

1 

1 

1 

1 

■  Extremely  Worthless 

■  Somewhat  Worthless 

■  Neutral 

■  Somewhat  Valuable 

■  Extremely  Valuable 


Comments  from  particpants: 


•  “The  introduction  was  not  interesting  or  valuable.” 


Question  10:  The  process  and  activities  used  in  the  workshop  significantly  enhanced 
the  creativity  of  the  group’s  ideas. 

Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  11:  The  overall  process  flowed  well. 


Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  12:  Rate  the  length  of  time  spent  in  each  of  the  following  areas  during  the 
workshop. 
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Question  13:  The  group  took  full  advantage  of  the  activities  introduced  during  the 
workshop. 


Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  14:  Rate  the  level  of  “structure”  for  this  workshop. 


Just  right 


Too  little 


Too  much 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  15:  OveraM,  the  workshop  facilitators  were  effective. 


Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  16:  The  workshop  facilitators  had  sufficient  knowledge  to  instruct  our 
group  on  the  process  used  during  the  workshop. _ 
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Question  17:  The  workshop  facilitators  answered  questions  effectively  during  the 
workshop. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  18:  Describe  your  area  of  specialization  as  compared  to  the  other  members 
of  the  workshop. 


Very  Similar  to  Mine 


k 


Somewhat  Similar  to  Mine 


Somewhat  Different  from  Mine 


Very  Different  from  Mine 


Other  (Please  Explain) 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Comments  from  participants: 

•  “Some  were  very  different  from  mine.” 
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Question  19:  I  had  a  great  deal  of  prior  knowledge/expertise  on  the  topic/problem  we 
examined  in  this  workshop. _ 


Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  20:  The  knowledge  and  experience  of  the  workshop  participants  were 
sufficient  to  explore  this  problem.  _ 


Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  21:  The  ideas  generated  during  the  workshop  were  valuable. 


Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  22:  The  workshop  gave  me  the  opportunity  to  express  my  ideas^ 

Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  23:  At  one  or  more  points  during  the  workshop,  the  group  approached  the 
problem  in  a  unique  way. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 

Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  24:  In  looking  back,  I  believe  there  were  ideas  that  our  group 
overlooked/dismissed. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Comments  from  participants: 

•  "Always  possible  to  find  new  approaches  and  thoughts.” 

Comments  from  POC: 

•  "Very  broad  topic;  hard  to  cover  it  all.” 
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Question  25:  We  accomplished  our  primary  purpose. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  26:  How  satisfied  were  you  with  your  experience? 

Completely  Satisfied 


Mostly  Satisfied 

Neither  Satisfied  nor  Dissatisfied 


Mostly  Dissatisfied 

Completely  Dissatisfied 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  27:  I  would  like  to  participate  in  similar  projects  in  the  future. 


Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Comments  from  POC: 

•  "Depending  on  topic.” 


Question  28:  In  future  workshops,  I  would  recommend  (check  all  that  apply 


Comments  from  participants: 


•  "Better  define  what  issues  and  situations  need  to  be  addressed.” 

•  “I  thought  it  was  all  just  right.” 

•  “Very  few  changes.  Good  group  size — functional.” 

•  "Right  amount  of  time  in  each  task/section 
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Comments  from  POC: 


“All  seemed  very  good  for  this  workshop.” 


Question  29:  Did  the  location  of  the  workshop  meet  your  needs? 


Yes 


Somewhat 


No  (Please  Explain) 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  30:  What  activities  did  you  like  most/least? 

Which  activities  did  you  like  most/least? 

Participant  Comments: 

MOST 

•  “Open  discussion,  small  group  activities,  zoom/re-zoom” 

•  “Small  group/word  association/analyses  of  topics  and  ideas” 

•  “I  enjoyed  every  exercise;  zoom/re-zoom  was  interesting” 

•  “I  liked  learning  about  each  other  and  the  break-out  sessions.” 

•  “Zoom/re-zoom” 

•  “Zoom/re-zoom  was  an  excellent  exercise  on  teamwork,  collaboration, 
perspective” 

•  “Zoom” 

•  “Zoom/re-zoom” 

•  “Zoom/re-zoom;  whole  group  worked  together  as  a  team;  respectful  of  one 
another-  able  to  complete  the  task  in  an  orderly  manner;  reached  goal” 

•  “Zoom/re-zoom;  introductions,  metaphoric  thinking  and  word  associations” 
LEAST 

•  “Converge  did  not  follow  small  group  time.  Ideas  were  lost  by  not  having  smaller 
groups  generate  thoughts.” 

•  “Diverge  break-out” 

POC  Comments: 

•  “All  seemed  important-  good  blend  of  small  group  and  large  group  activities.” 

•  “Zoom/re-zoom  and  getting  to  know  the  team;  I  would  have  liked  one  more  small 
group.” 
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Question  31:  What  was  your  overall  impression  of  the  Divergent  Collaboration81 

Workshop? 

Participant  Comments 

•  “I  am  extremely  impressed.  This  has  been  a  fascinating  day  and  a  half.  Though  I 
didn’t  know  what  to  expect  coming  in,  I  leave  with  a  high  regard  for  the  process  of 
Divergent  CollaborationSM  and  the  power  of  this  process.” 

•  “Love  different  points  of  view  looking  at  situations/seeing  how  they  tie  together 
and  add  my  original  thoughts/perspectives.  Very  interesting.” 

•  “I  thought  the  workshop  was  very  effective;  however,  I  think  it  is  too  short  to 
propose  ideas  to  such  a  complex  issue.  Would  have  liked  to  have  more  time  to 
discuss  ideas-  also  would  have  liked  to  have  known  specifically  what  solutions 
have  been  attempted  to  avoid  duplication  or  improve  existing  potential  ideas” 

•  “I  thought  it  all  worked  together  great.  Good  pacing  of  group  building  and 
exercises.  Great  facilitating  and  keeping  on  task  and  “in  the  moment.”  Enjoyed 
break-outs  with  different  people,  different  facilitators  would  have  been  good  for 
me  as  well  (I  had  Rachael  both  times).  Great  process  and  I  enjoyed  a  “cutting 
edge”  experience.” 

•  “Very  interesting  method  and  idea  of  bringing  various  backgrounds  together. 

Could  have  lots  of  potential  in  helping  some  organizations.” 

•  “I  enjoyed  the  group,  process  and  topic.” 

•  “Interesting,  fun,  educational,  free-spirited,  diverse  group,  accepting,  encouraging, 
out  of  the  box.” 

•  “It  was  a  fascinating,  stimulating  experience.  I  could  not  gain  a  sense  of  whether 
we  provided  value  to  AFRL,  but  it  certainly  was  a  valuable  experience  for  me.” 

•  “This  was  a  very  well  thought-out,  structured,  and  executed  workshop  that  I  feel 
very  lucky  to  have  been  able  to  take  part  in.  It  was  mutually  beneficial  for  most 
participants  because  we  all  learned  techniques/gained  advice  from  other 
participants,  and  we  were  able  to  provide  “out-there”  concepts/ideas  to  AFRL.” 

•  “I  enjoyed  working  on  the  project.  It  allowed  me  to  look  at  things  from  my 
perspective  as  well  as  think  outside  of  the  box.  Realizing  that  things  one  person 
does  on  a  daily  basis  that  is  routine  to  him/her  can  actually  be  beneficial  to 
someone  else  in  a  totally  different  field.  I  love  the  concept  of  this  type  of 
collaboration.” 

•  “The  concept  was  impressive.  To  be  open  to  utilize  very  different  areas  of 
expertise  and  experience  to  solve  specific  problems  is  something  not  normally 
done.  This  is  a  great  example  for  other  industries  to  use  the  concept  to  solve 
various  internal  problems  by  utilizing  unrelated/related  sources.” 

POC  Comments: 
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•  “Came  into  it  without  much  knowledge  of  the  process,  and  a  general  dislike  of 
thing  like  this,  but  walked  away  with  a  very  positive  experience.  Very  good 
method  for  thinking  outside  the  box  and  would  recommend  for  future  programs.” 

•  “Group  was  open  and  helpful,  communicated  well  and  respectful.  Team  leaders 
guided,  translated,  and  facilitated  excellently.  We  came  together  as  a  diverse 
group  to  address/discuss  a  problem  on  HPA.” 


Question  32:  What  changes  could  we  make  to  improve  future  projects? 

Participant  Comments: 

•  In  hindsight  I  realize  an  observer  might  learn  as  much  from  the  process  as  from 
what  he  produced.  Much  is  to  be  gained  for  what  is  learned  about  how  our 
divergent  group  worked  together.  This  alone  has  value.  Perhaps  we  all  had  a 
“product-centered”  expectation,  so  it’s  tough  to  know  how  successful  we  were. 
However,  I  also  hope  something  was  gained  by  how  our  group  approached  the 
process.” 

•  “Clearly  define  issue  or  situation  at  convergence  part  of  workshop.  What  AFRL  is 
trying  to  address  with  HPA  still  not  clear  to  me.  Hard  to  suggest  new  thoughts 
when  I  don’t  know  the  issue.” 

•  “Need  more  time  to  get  into  the  problem.  Two  days  for  Group  Dynamics  and 
understanding  Divergent  Collaboration811  and  three  days  to  get  into  the  Idea 
Phase-  should  propose  ideas  for  each  human  context.  For  example,  Day  1 : 
Intro/Group  and  Team  Exercise,  Day  2:  Explain  and  Experience  Divergent 
Collaboration^,  Day  3:  Social/Tech/Biological,  Day  4: 
Cognitive/Physical/Environmental,  Day  5:  Spiritual,  wrap-up,  and  admin” 

•  “None  come  to  mind  at  this  time.” 

•  “You’re  doing  very  well.  I  really  liked  the  way  Bart  and  Candace  interact.  More 
racial/ethnic  diversity  would  be  an  addition.” 

•  “After  problem  evaluation,  team  building,  etc.  Might  spend  a  bit  more  time  on 
solution  discussion  and  focus  a  bit  more  to  achieve  value  to  AFRL.” 

•  “Maybe  having  the  team  members  develop  a  “topic  sentence”  for  relating 
themselves/their  careers  together  to  show  the  overarching  commonality 
(experiences).  Keeping  the  group  better  focused  and  quicker  recognition  of  when 
the  evaluation  of  ideas  started  occurring.  Making  sure  introverted  people  are  truly 
getting  heard.” 

•  “During  the  break-out  sessions,  I  was  with  the  same  facilitator  each  time.  May 
help  increase  creativity  if  the  individual  groups  have  different  people  as  well  as 
different  facilitators.” 

•  “Continue  to  think  outside  the  box  in  selection  of  participants  and  divergent  areas 
of  profession,  expertise,  and  experiences.” 

POC  Comments: 

•  “Make  an  attendance  list  with  very  brief  bios  to  help  make  connections.” 

•  “I  hope  ideas  are  captured  so  we  may  be  able  to  use  them.  Stop  us/change  us 
from  a  specific  application/conop  sooner.  Stop  us  from  talking  too  long  on  one 
application.  Steer  us  to  wild  thinking.  Overall  group  seemed  conservative.” 
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Question  33:  Comments  and/or  Suggestions. 

Participant  Comments: 

•  “There  are  many  applications  for  those  types  of  events  in  our  business,  academic, 
artistic,  and  scientific  communities.  The  WBI  was  unknown  to  me  before  this 
point.  I  hope  to  work  with  you  again,  not  only  in  this  context,  but  in  the  context  of 
other  organizations  in  the  Dayton  community  as  well.  Thank  you!” 

•  “Thanks!” 

•  “Exciting  to  be  a  part  of  our  military  process.  This  was  very  enlightening  to  me  as 
a  US  citizen  and  tax  payer.  Education  to  the  population  (general  population)  is 
helpful  in  the  long  run  for  support  of  projects  and  research.  Much  is  publicized 
about  research  that  is  perceived  to  be  wasteful  and  frivolous  in  our  government.  I 
felt  true  value  to  our  country  and  servicemen  and  women  during  this  time.  My 
awareness  increased;  I  hope  the  communication  throughout  the  ranks  happens  at 
times  also,  i.e.,  the  folks  staring  at  the  screen  know  they  are  important  and 
research  is  being  done  to  help  their  performance  and  therefore  them  personally 
as  well.  Thanks  for  the  opportunity  to  be  a  part  of  this  awesome  group.” 

•  “Will  we  find  out  if  RH  gained  anything  valuable  from  this  workshop?” 

•  “Keep  up  the  good  work.  I’d  like  to  be  involved  with  the  Idea  Lab  in  the  future  as 
a  participant  and/or  as  a  facilitator.  I  like  the  model/process,  and  I  like  being 
involved  in  the  mission.  Thanks  for  involving  me.” 

•  “Great  job!  Ultimately,  I  appreciated  the  openness  of  the  environment/atmosphere 
to  allow  for  all  voices  to  get  heard  and  ideas  to  flow  freely/unabashed.” 

•  “The  facilitators  were  great,  very  knowledgeable,  and  able  to  direct  the  group  to 
meet  the  task  at  hand.  I  love  this  approach,  and  I  will  use  it  in  different  areas  of 
my  life.  Thank  you  for  this  opportunity.” 

•  “On  the  first  day  or  early  second  day  request  ‘best  practices’  info  in  writing  from 
participants.  Solicit  real  world  examples  of  related  application  on  the  given  topic. 
This  might  capture  some  good  insight  that  might  be  hesitant  to  be  verbally  shared 
in  a  group.  Any  that  might  apply  to  the  researchers’  issue  can  be  further  explored 
with  the  individual.” 

POC  Comments: 

•  “Great  food!  Team  was  extremely  professional,  friendly,  and  organized.  They 
respected  time  and  people.  Nice  group.” 
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7.10.  Appendix  J  -  Summary  of  HPA  Product  Questionnaires 


Question  1:  The  Workshop  helped  me  generate  new  insights  into  the  HPA  challenge 


l  Exec  Team 

AFRLHPA  Team 


Question  2:  At  one  or  more  points  during  the  workshop,  the  problem  space  was 
explored  in  a  unique  way. _ _ 
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Question  3.  There  were  new  researchjdeas  generated  from  the  workshop^ 


Question  4:  Please  assess  the  uniqueness/creativity  of  the  workshop  outcomes. 


Outcomes  were  highly 
unique/creative 

Outcomes  were  above 
average  in... 

Outcomes  were  moderately 
unique/creative 

Outcomes  showed  little 
uniqueness/creativity 

Outcomes  were  not 
unique/creative 

N/A  The  group  did  not 
propose  outcomes 
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Question  5:  The  workshop  helped  me  uncover  key  areas  for  investment. 


Completely  Agree 

Mostly  Aagree 

Neither  Agree  nor  Disagree 

■  AFRL  HPA  Team 

Mostly  Disagree 

Completely  Disagree 

0% 

0 

%  2C 

%  4C 

%  6C 

%  8C 

%  10 

Mote:  Only  the  AFRL  HPA  Team  filled  out  this  question  because  it  was  non-applicable  to 
the  Executive  Team. 

Comments: 

•  "Not  sure  yet--  certainly  many  things  to  think  about" 

Question  6:  I  could  have  achieved  the  same  or  similar  results  without  the  Divergent 
Collaboration91  Workshop. 

Completely  Disagree 

Mostly  Disagree 

Neither  Agree  nor 
Disagree 

Mostly  Agree 

Completely  Agree 


0%  20%  40%  60%  80% 

Note:  Only  the  AFRL  HPA  Team  filled  out  this  question  because  it  was  non-applicable  to 
the  Executive  Team. 


■  AFRL  HPA  Team 


100% 
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Question  7:  There  was  value  in  the  divergent  part  of  the  process. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


Exec  Team 


AFRLHPA  Team 


0%  20%  40%  60%  80%  100% 


Comments  from  HPA  Team: 

•  "Mostly  agree,  but  on  a  personal  level,  completely  agree" 


Question  8:  The  divergent  part  of  the  workshop  impacted  the  innovativeness  of  the 
products. 


Completely  Agree 


MostlyAgree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


Exec  Team 


AFRLHPA  Team 


0%  20%  40%  60%  80%  100% 
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Question  9:  The  workshop  helped  me  broaden  my  definition  of  HPA. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


Exec  Team 


AFRLHPA  Team 


0%  20%  40%  60%  80%  100% 


Question  10:  The  workshop  helped  me  think  about  HPA  in  a  different  way. 
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Question  11:  The  workshop  results  will  impact  my  future  or  existing  research 
activities. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


AFRLHPA  Team 


0% 


20%  40%  60%  80%  100% 


Mote:  Only  the  AFRL  HPA  Team  filled  out  this  question  because  it  was  non-applicable  to 
the  Executive  Team. 

Comments: 

•  "Not  sure  yet" 


Question  12:  I  would  use  Divergent  Collaboration8"  again. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


■  Exec  Team 

■  AFRLHPA  Team 


0%  20%  40%  60%  80%  100% 
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Question  13:  I  would  recommend  a  Divergent  Collaboration**  workshop  to  other 
AFRL  researchers. 


Completely  Agree 

Mostly  Agree 


Neither  Agree  nor  Disagree 

Mostly  Disgree 


Completely  Disagree 


Exec  Team 


AFRLHPA  Team 


0%  20%  40%  60%  80%  100% 


Question  14:  What  was  your  overall  impression  of  the  Divergent  Collaboration** 

Workshop? 

Comments  from  Exec  Team: 

•  "I  thought  the  workshop  was  significantly  improved  from  the  first  one  on 
Resiliency.  The  group  was  diverse  in  the  right  amount  (if  that  makes  sense),  and 
the  group  cohesion  was  strong,  nearly  from  the  very  beginning.  Very  happy  with 
it!  Loved  having  the  POC  support  we  did  (Luginbuhl)  and  thought  Nancy  and 
Josh's  participation  was  also  the  right  amount.  Much  better  ending/wrap-up  and 
better  results! 

•  "The  workshop  was  a  delight  to  participate  in—  it  had  a  lot  of  positive  energy 
surrounding  it  and  that  showed  in  the  final  product(s)" 

•  "The  workshop  was  much  more  successful  than  Resiliency;  more  group 
cohesion/smoother  'plan'/better  idea  generation" 

•  "The  workshop  was  full  of  ideas  and  discussion.  I  thought  that  there  was  a  good 
information  exchange  between  AFRL  and  the  participants.  People  were  making 
natural  connections  while  enhancing  creativity" 

•  "Terrific!  Best  yet." 

Comments  from  HPA  Team: 

•  "I  liked  the  process.  It  was  good  to  hear  other  peoples'  perspectives  maybe 
simpler  solutions  (non-tech)  to  linking  to  operational/performance  improvement.  I 
liked  the  discussion  group,  team  motivations,  intrinsic,  extrinsic  motivations" 

•  "Very  interesting  way  of  generating  new  ideas  and  assess  current  ideas.  Way 
outside  of  my  comfort  zone  but  was  extremely  helpful" 

Question  15:  What  changes  could  we  make  to  improve  future  workshops? 
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Comments  from  Exec  Team: 

•  "I  think  we  could  do  a  better  job  in  the  converge  section  and  application  to  the  real 
problem.  I  think  we  were  significantly  better  in  this  area  than  in  Resiliency,  but 
still  think  it  could  be  improved.  For  example:  more  succinct,  total  picture  of  the 
problem  (we  were  too  focused  on  the  "30  min  analyst")  and  being  able  to  facilitate 
the  group  seeing  the  bigger  picture.  Also  thought  the  matrix  used  in  the  converge 
section  wasn't  ideal.  We  could  have  bucketed  it  into  the  3  categories  after  the 
fact  (way  out  there  ideas,  actionable  research,  products)  or  just  used  them  as 
prompts  for  the  participants  to  think  about-  not  as  a  way  to  categorize  or  frame 
our  thinking.  I  would  have  liked  to  see  us  explore:  1 .)  before  [missions  and  work] 
2.)  during  [in  progress]  and  3.)  after  [missions  and  work] 

•  "Change  it  to  one  full  day  followed  by  1/2  day." 

•  "This  workshop  has  improved  significantly  since  the  last  one;  however,  we  need 
to  rework  the  overall  structure.  We  need  to  decrease  the  amount  of  "diverge" 
time  and  increase  the  "converging."  While  Bart  and  Kathy  were  successful  in 
"wrapping-up"  the  workshop,  I  don't  think  the  participants  were  ready/they  still  had 
more  ideas  coming.  We  need  to  plan  this  part  out  better." 

•  "Hard  to  come  up  with  many  changes.  I  would  continue  to  keep  the  group  in 
metaphoric  thinking  as  long  as  possible.  Overall,  I  thought  the  workshop  was 
great." 

•  "Take  it  even  further  with  the  lessons  learned.  But  the  design  for  the  HPA  is 
definitely  the  starting  point  for  the  Autonomy  DC." 

Comments  from  HPA  Team: 

•  "Reel  the  discussion  in  when  it  gets  to  problem  solving;  small  groups  for 
convergence  and  ideas  of  technology;  more  wild  ideas" 

•  "no  improvements  that  I  can  think  of. 

Question  16:  Comments  and/or  Suggestions? 

Comments  from  Exec  Team: 

•  “Much  better  than  Resiliency  Workshop.  Was  it  the  participants?  The  order  of 
events?  The  topic?  Not  sure.” 

Comments  from  HPA  Team: 

•  "Great  group;  Great  Divergent  CollaborationSM  organization  team;  very 
professional/out-going/open.  Great  food.” 
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Gender 


•  Female 

•  Male 

Age: 

•  20-29 

•  30-39 

•  40-49 

•  50-59 

•  Over  60 

Education: 


5  (33%) 

10  (67%) 


3  (20%) 

1  (7%) 

6  (40%) 

1  (7%) 

4  (26%) 


Post  Master’s  Degree 

7 

(47%) 

Master’s  Degree 

2 

(13%) 

Undergraduate  Degree 

6 

(40%) 

High  School  Diploma 

0 

(0%) 
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7.12.  Appendix  L  -  Comprehensive  List  of  Ideas  from  the  Autonomy  Grou 

HUMAN 


•  Culture/simplicity/trust  -  find  the  common  ground 

•  Marketing  good  things  about  UAVs;  educate  the  public 

•  Generational  considerations  -  understand  their  core  values,  attitudes,  way  they  work, 
how  they  view  autonomy 

-  Boomers  (ages  48-65) 

-  Gen  X  (ages  31-47) 

-  Millennial  (ages  18-30) 

-  TRUST  -  who’s  not  trusting?  The  RPA  operators  (younger  people)  trust 
what  they  are  doing;  the  ones  at  the  top  making  the  decisions  (older 
people)  generally  don’t 

•  Reverse  mentoring:  have  younger  generations  teach/train/mentor  older  generation 
on  technology 

•  Comfort  with  the  machine  ->  creates  trust 

-  Consistent  interaction 

-  Training 

-  Experience 

-  Education 

-  Simulation 

-  Mentoring 

•  Recognize  there  will  be  fear  of  adoption  that  will  need  to  be  worked  through 
MACHINE 

•  Look  at  simple  solutions  -  reduce  complexity,  simplify  the  machine 

•  Kiss,  think  Basic 

•  Understand  the  culture  of  desired  autonomy 

•  Start  new  -  blank  page 

•  Make  it  to  fulfill  operator’s  requirements,  not  what  AFRL  wants  it  to  be 

•  Don’t  approach  RPAs  in  the  same  way  as  a  traditional  aircraft,  approach  it  differently 

-  Change  roles  -  enlisted  person  by  flyer 

-  Different  ways  to  use  manning-  not  traditional  way  of  using  aircraft.  Make 
autonomy/automation  capabilities  on  what  people  using  it  actually  need. 
Focus  on  requirement  needs  for  RPA  operators 

•  Approach  from  human  perspective 


Distribution  A:  Approved  for  public  release;  distribution  unlimited. 
88  ABW  Cleared  06/18/2013;  88ABW-20 13-2894. 

86 


•  Simplify  user  interface 

•  Get  people  to  realize  systems  are  safe 
TEAMING 

•  Best  practices  from  the  highest  performing  teams 

•  Continued  virtual  collaboration  on  the  problem  with  Divergent  CollaborationSM  group 

•  Work  directly  with  RPA  Operator  and  analyst  to  explore  problem  (or  mission 
architects)  -  get  their  perspective  to  solve  problems;  functional  decomposition 

•  The  Four  Levels  of  Collaboration  -  determine  what  type  of  teaming/level  of 
collaboration  is  appropriate  for  each  human-machine  team  or  each  autonomous 
system 

-  Communicate 

-  Coordinate 

-  Cooperate 

-  Collaborate 

•  Mentoring  process  needed  for  new  members;  supervision,  feedback 

-  Like  Blue  Angels  or  Thunderbirds 

-  Bring  in  the  best  people  and  give  them  the  best  training 

•  Standardization  can  be  used  for  creating  teams,  e.g.  National  Aerospace  System 

-  For  consistency,  for  training 

•  Train  on  how  to  be  a  team  -  how  do  you  interact  with  each  other? 

•  Is  there  shared  understanding  between  AFRL  TDs  (siloed)? 

IN 

•  Focus  on  mid-term  issues,  not  just  now  and  25  years  in  the  future 
AUTONOMOUS 

•  Balancing  risk  of  autonomy,  perception,  and  politics 

•  What’s  left  over  after  autonomy  -  for  the  person  to  exercise  good  judgment.  This  is  a 
whole  new  and  different  way  of  thinking  about  the  problem 

-  What  level  of  proxy/authority  to  give  the  automation? 

•  Human  intervention  is  key  -  is  the  solution  and  the  problem: 

-  How  to  ensure  human  can’t  turn  off  automation/autonomy  if  they  don’t 
accept  it  so  the  machine  can  do  what  it  is  supposed  to  do  vs.  when 
machines  fail,  humans  pick  up  the  slack  -  leaving  things  to  machine  goes 
wrong  more  often  if  the  human  isn’t  there  to  intervene 


SYSTEMS 
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•  Just-in-time  best  practices 

•  More  efficient  process 

•  Look  at  biological  systems  for  sensing/means  of  communication 

•  Use  current  systems  that  use  human  and  study  what  it  would  be  like  without  human 
(use  gaming  and  simulations) 

•  Use  best  practice  systems  that  exist  to  design  the  right  system  (capitalize  on  lessons 
learned) 

•  Safety  systems  -  market  it 

•  “Should  we  be  doing  this?”  -  employ  process  engineer  to  look  at  the  actual  process. 
What  can  we  avoid  doing? 

•  AF  operates  in  silos;  TDs  don’t  work  together  -  not  systems  engineering  approach 
(AF  culture) 
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7.13.  Appendix  M  -  Summary  of  Autonomy  Process  Questionnaires 


Question  1:  I  came  to  the  workshop  with  clear  expectations. 


Completely  Agree 


k 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Comments  from  participants: 

•  "Mostly  disagree,  but  that's  OK!" 

•  "I  think  not  knowing  was  on  purpose." 

•  "It  was  clear  you  weren't  telling  us." 
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Question  3:  Terms  of  engagement  were  clear. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  4:  Everyone  participated  in  the  group  discussion  during  the  workshop. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  5:  I  felt  constrained  to  only  pursue  certain  types  ofjdeas. 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 

Mostly  Disagree 


Completely  Disagree 


Question  6:  The  discussions  during  the  workshop  were  valuable. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  7:  The  process  and  activities  used  in  this  workshop  were  essential  in 
helping  our  group  meet  its  objective^ _ _ _ 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  8:  The  Process  Diagram  helped  me  to  track  progress  during  the  workshop. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  9:  Rate  the  activities  below  in  terms  of  if  you  think  they  were  valuable  to 
the  workshop. 


100% 

90% 

80% 

70% 

60% 


50% 

40% 

30% 


■  Extremely  Worthless 

■  Somewhat  Worthless 

■  Neutral 

■  Somewhat  Valuable 

■  Extremely  Valuable 


Question  10:  The  process  and  activities  used  in  the  workshop  significantly  enhanced 
the  creativity  of  the  group’s  ideas. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Comments  from  participants: 

•  "Homework  not  detailed  enough" 
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Question  11:  The  overall  process  flowed  well. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  12:  Rate  the  length  of  time  spent  in  each  of  the  following  areas  during  the 
workshop. 


inno/ 

±uu/u 

Q  fiO/ 

1 

jU/o 

ou/o 

700/ 

/U/o 

ou/o 

CrtQZ 

jU/o 

1 

4U/0 

1 

ll 

jU/o 

jno/ 

1 

1 

II 

■  Too  Much  Time 

ZU/o 

1 

1 

1 

■  Too  Little  Time 

±U/o 

rtOZ 

j 

1 

■ 

■ 

J 

■  IIH1I 

U/o 

/ 

1  1  1  1  1  1  1  1 

„eS  >  <3-  J*  > 

^  s  /  /  /  /  /  / 
/  /  /  f  s  /  * 

^  ^  -V°  <5S  ^S-° 

■  Just  Enough  Time 

■  N/A 
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Question  13:  The  group  took  full  advantage  of  the  activities  introduced  during  the 
workshop. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 

Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  14:  Rate  the  level  of  “structure”  for  this  workshop. 


Just  right 


Too  little 


Too  much 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Question  15:  OveraM,  the  workshop  facilitators  were  effective. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  16:  The  workshop  facilitators  had  sufficient  knowledge  to  instruct  our 
group  on  the  process  used  during  the  workshop. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Comments  from  participants: 

•  "more  interaction  between  the  main  facilitators  and  the  speakers  (Kris/Bob)  would 
have  been  beneficial  in  this  session” 
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Question  17:  The  workshop  facilitators  answered  questions  effectively  during  the 
workshop. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Comments  from  participants: 

•  "Each  one  answered  but  together  were  incoherent.” 

Question  18:  Describe  your  area  of  specialization  as  compared  to  the  other  members 
of  the  workshop. _ 


Very  Similar  to  Mine 


Somewhat  Similar  to  Mine 


Somewhat  Different  from  Mine 


Very  Different  from  Mine 


Other  (Please  Explain) 
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Question  19:  I  had  a  great  deal  of  prior  knowledge/expertise  on  the  topic/problem  we 
examined  jn  this  workshop. _ 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  20:  The  knowledge  and  experience  of  the  workshop  participants  were 
sufficient  to  explore  this  problem.  _ 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Comments  from  participants: 

•  “depends  on  definition  of  the  problem” 


Distribution  A:  Approved  for  public  release;  distribution  unlimited. 
88  ABW  Cleared  06/18/2013;  88ABW-20 13-2894. 

98 


Question  21:  The  ideas  generated  during  the  workshop  were  valuable. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  22:  The  workshop  gave  me  the  opportunity  to  express  my  ideas^ 

Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 

Comments  from  participants: 

•  “except  when  some  took  over  the  discussion  in  the  break-out  groups” 
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Question  23:  At  one  or  more  points  during  the  workshop,  the  group  approached  the 
problem  in  a  unique  way. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Comments  from  participants: 

•  “Nathan’s  “why”  question  ...  but  little  time  spent  on  it.” 

Question  24:  In  looking  back,  I  believe  there  were  ideas  that  our  group 
overlooked/dismissed. 


Completely  Agree 

■ 

Mostly  Agree 

Neither  Agree  nor  Disagree 

Mostly  Disagree 

Completely  Disagree 

0% 

0 

%  1C 

%  2C 

%  30 

%  4C 

%  5C 

%  60 

%  70 

%  80 

%  90 

%  10 

Comments  from  participants: 

•  “Nathan’s  “why”  question  ...  but  little  time  spent  on  it.” 
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Question  25:  We  accomplished  our  primary  purpose. 


Completely  Agree 


k 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Comments  from  participants: 

•  “N/A,  uncertain” 

•  “not  quite  sure  what  our  primary  purpose  was” 


Question  26:  How  satisfied  were  you  with  your  experience? 


Completely  Satisfied 


Mostly  Satisfied 


Neither  Satisfied  nor  Dissatisfied 


Mostly  Dissatisfied 


Completely  Dissatisfied 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Comments  from  participants: 

•  “was  expecting  to  accomplish  more,  left  unable  to  contribute  enough” 
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Question  27:  I  would  like  to  participate  in  similar  projects  jn  the  future. 


Completely  Agree 


Mostly  Agree 

Neither  Agree  nor  Disagree 


Mostly  Disagree 

Completely  Disagree 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Question  28:  In  future  workshops,  I  would  recommend  (check  all  that  apply 


Comments  from  participants: 

•  “!!!” 

•  “Gauge  amount  of  teamwork  necessary” 

•  “longer  discussion” 
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Question  29:  Did  the  location  of  the  workshop  meet  your  needs? 


Yes 


Somewhat 


No  (Please  Explain) 


0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Comments  from  participants: 
•  “cold  though” 


Question  30:  What  activities  did  you  like  most/least? 

Which  activities  did  you  like  most/least? 

Participant  Comments: 

MOST 

•  “Diverge/converge... the  activities  where  we  could  list  the  random  things  that 
came  to  mind. 

•  “Brief  out” 

•  “Breakouts” 

•  “Liked  the  zoom  activity;  valued  the  discussions” 

•  “Breakouts,  Converge” 

•  “Break  out  session  -  coming  back  together  as  a  larger  group” 

•  “Introduction,  network,  getting  to  know  participants,  breakouts,  Bart’s  recap  of  the 
1 .5  days” 

•  “The  homework  discussion;  Kris’  slide  presentation  at  the  end  of  day  1” 

•  “I  liked  the  breakout  sessions  most” 

LEAST 

•  “The  AFRL  problem  activity  seemed  very  specific  but  unclear  as  to  what 
specificity  the  subject  was.” 

•  “What  I  liked  least  was  not  having  access  to  the  big  picture  and  allowing  each 
team  member  to  decide  what  the  problem  is.  Not  having  a  clear  definition  of  what 
the  “perceived”  problem  is  and  allowing  us  to  ascertain  what  the  “real”  problem 
may  be” 

•  “Would  have  much  preferred  a  little  more  discussion  of  the  actual  problem  and  its 
requirements  and  constraints.  Allow  the  teams  to  decide  what  the  real  problem  is 
from  each  of  their  perspectives!  Do  not  assume  we  will  be  biased  because  we 
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know  how  the  problem  was  solved  in  the  past.  Let  us  understand  the  entire 
problem  from  the  highest  possible  level  and  then  allow  us  to  expand  and  explore 
the  boundaries.” 

•  “Defining  human-machine-teaming  as  a  group” 

•  “Defining  Human-Machine  teaming.  I’m  not  sure  we  ever  came  up  with  an 
answer.  I  could  guess  we  might  have  several  different  answers  in  our  group.” 

•  “Connecting  by  yarn,  introducing  each  other  instead  of  ourselves” 

•  “String  thing” 

•  “Did  not  dislike  anything.  Need  to  get  to  know  each  other  to  feel  comfortable  with 
each  other  is  important  for  many  people.” 

POC  Comments: 

•  “Least  favorite  activity  was  the  String  Exercise  -  not  certain  we  really  got  how 
people  were  connected  -  that  understanding  didn’t  really  occur  until  analogous 
activity  discussion  on  Day  2.” 

Question  31:  What  was  your  overall  impression  of  the  Divergent  Collaboration8" 

Workshop? 

Participant  Comments 

•  “It  was  very  interesting  meeting  and  working  with  completely  different  people.  It 
was  pretty  confusing  and  I  think  working  with  that  confusion  was  part  of  the 
process  and  challenge.” 

•  “Outstanding  experience.  I  believe  I  got  more  out  of  this  than  you!” 

•  “Good,  but  slightly  incoherent;  the  facilitators  had  difficulty  conveying  a  coherent 
and  consistent  problem-space  and  goals  for  the  groups.” 

•  “It  was  very  interesting.  I  think  people  were  different  but  am  not  sure  that  there 
was  a  lot  of  “divergent”  thinking.  Thought  we  might  get  to  offer  more  unique 
outside  the  box  thinking.  But  I  think  this  would  require  a  different  type  of  early 
workshop  training.” 

•  “Useful  for  topics  where  expertise  or  maturity  of  area  leads  to  limited/insular 
thinking.” 

•  “Bright,  thoughtful  participants  and  skilled  facilitators.” 

•  “It  was  hard  to  stay  in  the  problem  space  for  several  reasons:  1 .  There  were 
many  problem  solvers  in  the  group,  2.  We  were  not  sure  what  the  problem  was. 
We  were  trying  to  define  problems  for  ??” 

•  “Interesting  concepts  presented.  Overall  some  of  the  best  group 
workshops/discussions  I’ve  been  involved  with.” 

•  “I  think  the  process  is  very  beneficial,  but  it  may  require  participants  to  complete 
more  than  one  workshop  to  be  more  effective  and  feel  more  comfortable  with  the 
process  and  to  be  a  bit  more  collaborative.” 

•  “It  was  much  better  than  I  expected.  The  information  and  dynamics  created  by 
the  mixed  group  leveraged  knowledge.” 
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•  “I  learned  much  more  than  what  ever  did  about... not  only  UAV’s...but  also  how 
complicated  the  process  of  working  with  the  governments  entity  can  be... since 
this  was  my  first  opportunity  to  participate  in  such  a  session.” 

•  “I  think  it  was  somewhat  self-fulfilling.  Looking  for  more  weeds. .  .not  more  weed- 
whacking.” 

•  “I  liked  the  idea  of  not  completely  specifying  the  problem  initially  and  creating 
some  anxiety  for  some  team  members  (especially  me),  however,  I  felt  we  needed 
a  clearer  description  of  the  actual/perceived  problem.  I  totally  agree  with  the  idea 
of  having  very  diverse  groups  of  thinkers/participants.  Loved  the  teambuilding 
exercises,  however,  I  would  have  gone  one  step  further  wrt  taking  the  group  to  an 
extremely  absurd  exercise  (e.g.,  at  our  last  MIT  meeting  we  had  subject-matter- 
experts,  faculty  and  MIT  grad/undergrad  students  perform  a  flashmob  at  noon  in 
the  MIT  Student  Center  of  a  Haka  (http://www.youtube.com/watch?v=c-lrE2JcQ44  as 
soon  as  we  have  the  MIT  version  I  will  forward  if  you  like).  I  think  you  definitely 
have  a  process  which  has  worked  for  you  in  the  past  and  which  is  based  upon 
many  years  of  creating  teams  and  solving  very  complex  problems,  however,  I  felt 
that  this  workshop  was  focused  on  only  one  specific  phase  of  that  process  and  we 
were  not  supposed  to  have  access  to  and/or  participate  in  the  full/complete 
process.  I  believe  that  was  what  caused  the  most  anxiety  for  me.” 

POC  Comments: 

•  “I’m  not  certain  that  this  topic  was  suited  for  this  type  of  workshop.” 


Question  32:  What  changes  could  we  make  to  improve  future  projects? 

Participant  Comments: 

•  “Too  much  emphasis  was  placed  on  specific  machines  and  a  seemingly  specific 
problem — but  then  we  were  told  not  to  think  about  the  problem.  Maybe  asking 
each  person  to  think  about  a  problem/solution  that  they  have  encountered  in 
his/her  own  occupation — and  then  draw  aspects  from  those  descriptions  instead 
of  talking  about  a  problem  and  asking  how  your  experience  relates  to  it...” 

•  “None.  Facilitators  were  outstanding!!” 

•  “Facilitators  need  a  better  understanding  or  better  means  of  conveying  what  they 
want.  I  don’t  know  how  precisely  but  have  a  few  thoughts.  It’s  almost  as  if  you 
needed  a  workshop  to  define  this  workshop?” 

•  “You  gave  one  exercise  to  make  the  team  work  together.  Think  about  using 
exercises  related  to  paradigm  shift.  Thinking  outside  the  box  types  of  exercises.” 

•  “Workshop  topic  perhaps  needs  to  be  expressed  as  a  question  and  not  as  a 
theme.” 
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•  “Be  more  assertive  in  facilitating  group  discussions  to  ensure  input  by  all  and  less 
dominance  by  a  few.  Provide  directory  of  participants’  contact  info.” 

•  “More  clearly  define  what  you  are  trying  to  get  out  of  the  workshop.” 

•  “Better  define  the  problem  or  purpose  of  the  workshop.  This  would  be  especially 
helpful  for  the  homework  part.  People  seemed  to  be  prepared  for  the  wrong  thing 
when  they  showed  up  in  the  morning.” 

•  “I  understand  why  the  “problems”  were  not  detailed  initially  but  greater  context — 
and  who  was  involved  with  the  problem  from  the  group  would  have  been  helpful.” 

•  “Not  much,  I  have  nothing  else  to  compare  it  with.” 

•  “It  would  be  interesting  to  bring  similar  groups  together,  side-by-side,  on  the  same 
well  defined  problem.  One  group  using  Divergent  Collaboration811  and  one  using  a 
“solve-this!”  method.  Charlie  and  I  were  dying  to  attack  the  problem  ©.” 

•  “Add  one  or  two  more  days,  include  several  sessions  to  immerse  the  team  into 
what  the  “actual’7”perceived”  problem  is  that  you  are  trying  to  solve  (just  saying 
manpower/money  is  the  issue  and  a  very  brief  idea  of  what  the  problem  is  (RPAs 
surveilling/attacking/targeting/flying,  etc.)  was  not  enough  in  my  opinion.  Perhaps 
add  some  type  of  simulation  capability  for  participants  to  immerse  themselves  in  a 
game  which  simulates  the  actual  problem/activity.  I  would  have  loved  to  fly  a 
simulated  mission.  Which  reminds  me,  we  were  never  told  what  an  actual 
mission(s)  was!” 

•  “Really  did  not  like  Scott’s  answer  which  implied  that  we  should  not  worry  about 
the  technical  details  because  we  have  some  really  good  technical  folks  already 
solving  that  part  of  the  problem.  I  would  say  this  was  the  greatest  turnoff  for  me 
and  something  which  I  have  experienced  personally  totally  impedes  solutions  to 
problems.  It  is  that  kind  of  arrogance  (not  on  Scott’s  part)  which  allowed  two 
Shuttle  accidents  to  occur  and  which  could  have  prevented  several  of  the  teams  I 
worked  with  from  developing  the  innovations  we  eventually  flew.  I  never  believe 
the  “experts”  know  anything  completely  and  am  constantly  pushing  for  verification 
and  asking  my  teams  to  explore  the  impossible.  Not  allowing  the  diverse  team  to 
explore  every  aspect  of  the  problem  not  only  sends  a  message  that  their  input  is 
not  valued  (or  would  not  help  to  creatively  solve  part  of  the  problem)  but  that  they 
have  to  limit  their  ideas  to  a  much  smaller  section  of  the  problem  without 
understanding  the  complete  picture  and  how  their  ideas  could  interact  with  other 
aspects  of  the  system.” 

POC  Comments: 

•  “Probably  would  spend  more  time  up  front  with  topic  sponsor/owner  to  better 
understand  workshop  as  well  as  to  better  prepare  scoping  of  “the  problem.” 

Question  33:  Comments  and/or  Suggestions. 
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Participant  Comments: 

•  “Really  liked  small  group  work.” 

•  “See  me  over  a  beer.  Ideas  flow  better.” 

•  “Presentation  by  Kris  should  either  be  more  extensive  or  scrapped.  Just  confused 
people  who  weren’t  familiar  with  the  problems  facing  RPA.” 

•  “I  enjoyed  participating  and  I  hope  that  I  was  able  to  help  out  in  just  a  small  way.  I 
really  enjoyed  being  with  all  the  participants  and  have  a  great  deal  of  respect  for 
what  the  project  team  was  trying  to  accomplish.” 

•  “Thank  you.  ©” 

•  “I  think  I  exhausted  most  of  my  ideas  above;  however,  I  would  like  to  add  the 
inclusion  of  some  type  of  virtual  platform  for  collaboration  by  which  team  members 
could  continue  to  discuss  ideas,  learn  more  details  of  the  problem  and  perhaps 
help  influence  a  new  direction  of  thought,  concept,  etc.” 

POC  Comments: 

None 
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7.14.  Appendix  N  -  Summary  of  Autonomy  Product  Questionnaires 


Question  1:  The  Workshop  helped  me  generate  new  insights  into  the  Autonomy 
challenge. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


Question  2:  At  one  or  more  points  during  the  workshop,  the  problem  space  was 
explored  in  a  unique  way. _ 
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Question  3:  There  were  new  research  ideas  generated  from  the  workshop. 


Completely  Agree 


i 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disgree 


Completely  Disgree 


l  Exec  Team 


AFRL  Autonomy 
Team 


0%  20%  40%  60%  80%  100% 


Question  4:  Please  assess  the  uniqueness/creativity  of  the  workshop  outcomes. 


Outcomes  were  highly 
unique/creative 

Outcomes  were  above 
average  in... 


Outcomes  were  moderately 
unique/creative 

Outcomes  showed  little 
uniqueness/creativity 


Outcomes  were  not 
unique/creative 


N/A  The  group  did  not 
propose  outcomes 


0%  20%  40% 


60% 


■  Exec  Team 


■  AFRL  Autonomy 
Team 


80%  100% 
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Question  5:  The  workshop  helped  me  uncover  key  areas  for  investment. 


Completely  Agree 


Mostly  Aagree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


AFRL  Autonomy 
Team 


0% 


20% 


40%  60% 


80%  100% 


Mote:  Only  the  AFRL  Autonomy  Team  filled  out  this  question  because  it  was  non-applicable 
to  the  Executive  Team. 


Question  6:  I  could  have  achieved  the  same  or  similar  results  without  the  Divergent 
Collaboration31  Workshop. 


Completely  Disagree 


Mostly  Disagree 

Neither  Agree  nor 
Disagree 

Mostly  Agree 


■  AFRL  Autonomy 
Team 


Completely  Agree 


0%  20%  40%  60%  80%  100% 

Note:  Only  the  AFRL  Autonomy  Team  filled  out  this  question  because  it  was  non-applicable 
to  the  Executive  Team. 
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Question  7:  There  was  value  in  the  divergent  part  of  the  process. 


Completely  Agree 

Mostly  Agree 

Neither  Agree  nor  Disagree 

■  Exec  Team 

E  AFRL  Autonomy 

Team 

0% 

Mostly  Disagree 

Completely  Disagree 

0 

%  20 

%  40 

%  60 

%  80 

%  10 

Question  8:  The  divergent  part  of  the  workshop  impacted  the  innovativeness  of  the 
products. 


Completely  Agree 


MostlyAgree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


■  Exec  Team 

■  AFRL  Autonomy 
Team 
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Question  9:  The  workshop  helped  me  broaden  my  definition  of  Autonomy. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


Exec  Team 


AFRL  Autonomy 
Team 


0%  20%  40%  60%  80%  100% 


Question  10:  The  workshop Jielped  me  think  about  Autonomy  in  a  different  way. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disgree 
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Question  11:  The  workshop  results  will  impact  my  future  or  existing  research 
activities. 


Completely  Agree 


Mostly  Agree 


Neither  Agree  nor  Disagree 


Mostly  Disagree 


Completely  Disagree 


0% 


AFRL  Autonomy 
Team 


20%  40%  60%  80%  100% 


Mote:  Only  the  AFRL  Autonomy  Team  filled  out  this  question  because  it  was  non-applicable 
to  the  Executive  Team. 


Question  12:  I  would  use  Divergent  Collaboration81  again. 
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Question  13:  I  would  recommend  a  Divergent  Collaboration**  workshop  to  other 
AFRL  researchers. 


Question  14:  What  was  your  overall  impression  of  the  Divergent  Collaboration*1 

Workshop? 

Comments  from  Exec  Team: 

•  "Good  but  needs  to  be  tailored” 

•  "Autonomy  was  too  much  for  the  participants  to  handle  so  they  chose  their  own 
specific  problem  area  within  autonomy.  This  allowed  them  to  their  attention  on 
something  specific.  This  did  not  necessarily  line  up  with  the  POC’s  problem. 

Also,  the  POC’s  talk  seemed  way  too  technical  for  the  participants.” 

•  "Participants  thought  POC  wanted  them  to  solve  manning  issue  rather  than 
explore  possibilities.” 

Comments  from  HPA  Team: 

•  "The  workshop,  itself,  is  a  very  interesting  approach.  I  think  either  the  topic  of 
human/systems  teaming  for  Autonomous  systems  is  too  broad  or  possibly  the 
activities  could  have  been  directed  differently  to  improve  the  group’s  productivity.” 

•  "Probably  more  useful  for  a  mature  or  otherwise  “stovepiped”  area  where  ideas 
may  be  stagnating.” 

Question  15:  What  changes  could  we  make  to  improve  future  workshops? 

Comments  from  Exec  Team: 

•  "Look  at  each  DC  uniquely  and  develop  appropriate  processes  (not  just 
workshop)  for  each” 

•  "Have  a  definite  challenge  area” 

•  "Fewer  engineers  for  such  technical  issues” 

•  "More  break  out  groups” 

•  "more  informed  POC  (regarding  process)” 
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Comments  from  HPA  Team: 

•  "I’m  not  certain  that  the  group  needed  to  do  the  string  exercise.  Everyone  came 
ready  to  engage  so  I’m  not  certain  it  added  anything.  Possibly  spending  more 
time  prior  to  the  workshop  to  really  understand  and  develop  the  problem 
statement.  Use  more  prop  work  to  help  frame  the  problem  owner’s  briefing  at  the 
end  of  the  day.” 

Question  16:  Comments  and/or  Suggestions? 

Comments  from  Exec  Team: 

None 

Comments  from  HPA  Team: 

None 
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7.15.  Appendix  O  -  List  of  Acronyms 


AFRL 

DC 

DiSC 

HPA 

IDEA 

RH 

RY 

S&Es 

STEM 

WBI 


Air  Force  Research  Laboratory 
Divergent  Collaboration 

Dominance,  Influence,  Steadiness,  and  Compliance 

Human  Performance  Augmentation 

Innovate,  Demonstrate,  Explore,  Apply 

Human  Effectiveness  Directorate 

Sensors  Directorate 

scientists  and  engineers 

Science,  Technology,  Engineering,  and  Mathematics 
Wright  Brothers  Institute 
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